DEV Community: vishalmysore

PROOF for AI Agents: A Technical Scoring Rubric For Self Evaluation

vishalmysore — Wed, 19 Aug 2026 20:17:47 +0000

PROOF — Planning, Reasoning, Orchestration, Observability, Feedback — is a five-category, 25-point rubric for scoring whether an "AI agent" claim is actually backed by agentic architecture. The version below breaks each category into measurable sub-criteria instead of a single subjective 0–5 gut call, and applies it to two domains where the difference between a real agent and a well-dressed prompt isn't just an ROI question — it's a safety and liability question: healthcare and manufacturing.

The Rubric, Formalized

Each category is worth 5 points, built from concrete, checkable sub-criteria rather than a single impression score.

Planning (5 pts)

Explicit goal representation and task decomposition artifact exists — a plan graph, task tree, or structured intermediate representation, not just a prompt (0–2)
System can revise the plan mid-execution when new information invalidates the original approach, rather than executing a fixed script (0–2)
Constraint handling prunes invalid or unsafe plan branches before execution, not after (0–1)

Reasoning (5 pts)

Outputs are grounded in retrievable evidence — cited tool output, sensor data, or source documents — not just fluent text (0–2)
A verification or self-consistency pass exists (a second model, a rule-based checker, or a cross-reference step) before the reasoning is acted on (0–2)
The system can abstain or escalate under uncertainty instead of producing a confident guess (0–1)

Orchestration (5 pts)

Coordination runs on an explicit state machine or durable workflow engine (LangGraph, Temporal, a formal state graph), not ad hoc prompt chaining (0–2)
Each step has defined retry, rollback, and failure-handling behavior — not "if it errors, stop" (0–2)
Sub-agents or stages communicate through typed, validated interfaces (schemas), not free-text handoffs (0–1)

Observability (5 pts)

Per-step tracing — tool calls, latency, token/compute cost — is exported to a real system (OpenTelemetry, a trace store, a cost dashboard), not just visible in a terminal (0–2)
An immutable, exportable audit log exists, sufficient for a third party to reconstruct what happened and why (0–2)
Cost and usage are attributed per task or per case, not just aggregated monthly (0–1)

Feedback (5 pts)

An automated evaluation suite (golden test set, regression tests) runs pre-deployment and continuously, not just once at launch (0–2)
Irreversible or high-consequence actions are gated behind a human checkpoint, not auto-executed (0–2)
Flagged failures feed back into the system — an updated eval, a corrected prompt, a retraining trigger — closing the loop rather than being logged and forgotten (0–1)

Healthcare: Clinical Intake and Triage Support

Two versions of the same idea — an agent that reviews patient intake data and flags cases for clinician attention — scored against the same rubric.

Version A — the thin wrapper. A single LLM call takes free-text intake notes and outputs a "risk level" and a suggested next step, displayed directly to a nurse with no supporting detail.

Category	Score	Why
Planning	1/5	One-shot classification, no decomposition, no revision if new labs come in mid-session
Reasoning	1/5	Risk label isn't traceable to specific findings; asking "why" gets a plausible-sounding paragraph, not a cited data point
Orchestration	0/5	No pipeline — a single call in, a single output out
Observability	0/5	No trace of which fields the model weighted, no audit trail beyond the final label
Feedback	1/5	No eval suite; "verification" is whether the nurse happens to disagree and overrides it
Total	3/25	Bagent — regardless of the clinical framing, this is a single unverified LLM call with real-world consequences riding on it

Version B — the engineered version. Structured intake fields feed a decomposition step that checks against defined clinical decision rules and known red-flag combinations; every flagged risk factor links to the specific vital sign, lab value, or reported symptom that triggered it; the pipeline runs on a durable workflow engine with a retry policy per data source; every run is traced and logged to an audit store built for compliance review; nothing is escalated to "urgent" without a rule-based checker agreeing with the model's assessment, and any case the model is uncertain about is auto-routed to a clinician rather than guessed at; a continuously-run eval suite checks the system against a held-out set of historical, clinician-labeled cases before every deployment.

Category	Score	Why
Planning	4/5	Structured decomposition against defined criteria; revises assessment as new intake data arrives
Reasoning	4/5	Every flag traces to a specific data point; uncertain cases are surfaced, not guessed
Orchestration	5/5	Durable workflow engine, explicit retry/rollback per data source, typed interfaces between stages
Observability	4/5	Full per-step tracing and an audit-grade log; cost/latency attribution present
Feedback	5/5	Continuous eval against clinician-labeled cases, hard checkpoint before any "urgent" escalation, closed-loop correction when a miss is found
Total	22/25	Real Agent — genuinely agentic, and just as importantly, built so a human can audit exactly why it did what it did

The gap between these two isn't model quality — both could run on the same underlying LLM. It's whether Reasoning is traceable and whether Feedback includes a hard gate before anything irreversible happens. In a clinical setting, those are exactly the two categories liability actually hinges on.

Manufacturing: Production-Line Anomaly Response

Same exercise, applied to an agent monitoring sensor data on a production line and responding to anomalies — everything from a temperature drift to a full line stoppage.

Version A — the thin wrapper. A model watches a stream of sensor readings, and when something looks off, it calls an API to adjust a setpoint or halt the line, based on a prompt describing "normal" operating ranges.

Category	Score	Why
Planning	1/5	Reacts to the latest reading; no model of what "normal drift" vs. "genuine fault" looks like over time
Reasoning	1/5	No grounding beyond the instantaneous reading — no correlation against related sensors, maintenance history, or known fault signatures
Orchestration	1/5	Single call triggers a single action; no defined behavior if the halt command itself fails to execute
Observability	1/5	Action gets logged as an event, but no trace of what data or reasoning triggered it
Feedback	0/5	No eval against historical fault data; nothing checks whether the halt call was even the right call after the fact
Total	4/25	Bagent — a live model wired directly to a physical actuator with no verification step is a hazard, not an agent

Version B — the engineered version. Sensor streams feed a state model that distinguishes normal drift from anomaly using multi-sensor correlation and historical baselines; any proposed corrective action is checked against interlock and safety constraints before execution; the workflow runs on an orchestration layer with explicit rollback if an actuator command fails to confirm; every decision — inputs, correlated signals, the action taken, and the interlock check result — is traced to a time-series log tied to the plant's existing OPC-UA/historian infrastructure; low-confidence anomalies alert a human operator instead of triggering an automatic response, and full line stoppages always require operator confirmation regardless of model confidence; the fault-detection model is continuously evaluated against a labeled library of past fault events and re-validated whenever a new fault type is added.

Category	Score	Why
Planning	4/5	Multi-sensor correlation against historical baselines, not single-reading reaction
Reasoning	4/5	Every action traces to specific correlated signals; low-confidence cases don't get auto-actioned
Orchestration	5/5	Explicit interlock checks, confirmed rollback on failed actuator commands
Observability	5/5	Full decision trace tied into existing historian infrastructure — auditable by plant engineers, not just data scientists
Feedback	4/5	Continuous evaluation against a labeled fault library; hard human gate on full stoppages
Total	22/25	Real Agent — same category of task as Version A, built so a wrong call gets caught before it touches a physical process

What the Two Domains Have in Common

Line up both Version B scorecards and the pattern holds across completely different domains: the categories that separate a real agent from a dangerous wrapper aren't Planning or Orchestration — plenty of thin wrappers can fake a plausible-looking pipeline. It's Reasoning traceability and the Feedback hard-gate. In both healthcare and manufacturing, the real agent's defining trait isn't that it's smarter — it's that every consequential action is grounded in a specific, checkable piece of evidence, and nothing irreversible happens without either a rule-based check or a human confirming it first.

That's the actual engineering takeaway for anyone scoring an agent in a regulated or safety-critical domain: don't spend your review time asking whether the plan looks sophisticated. Spend it asking to see the trace for the last high-stakes decision it made, and asking who — or what — had to sign off before it acted.

Note: the scenarios above are illustrative composites built to demonstrate the scoring method, not descriptions of any specific deployed product. Nothing here is medical, safety, or compliance advice — treat it as an engineering evaluation framework, not a certification standard.

Is This a Real AI Agent? Ask for PROOF

vishalmysore — Wed, 19 Aug 2026 19:55:24 +0000

Every other post on the AI and engineering subreddits this year has an "AI agent" in it somewhere. Most of them can't survive one question: where's the PROOF? That's not rhetorical — PROOF is a five-part checklist you can actually run against any agentic AI claim, live, in a thread, a standup, or a demo, before anyone signs off on it.

The word "agent" has become one of the most overused terms in software, and the threads where practicing engineers actually compare notes — r/ExperiencedDevs, r/developersIndia, and similar corners of the internet — keep circling back to the same complaint. It's applied to everything from genuinely autonomous systems that plan, act, and self-correct with minimal supervision, to a single prompt with a role attached and a rebrand. The gap between those two things is enormous — in engineering complexity, in reliability, and most of all, in cost. And because the word itself has stopped meaning anything specific, most people evaluating an "agent" have no consistent way to tell which one they're actually looking at.

PROOF fixes that. It's five questions — Planning, Reasoning, Orchestration, Observability, Feedback — and a simple scoring method that turns "does this feel agentic?" into something you can actually answer.

Why "Agent" Stopped Meaning Anything

Part of the problem is that building something that looks agentic in a demo is easy. Wire a large language model to a couple of tools, give it a system prompt describing a role, chain two or three calls together, and you have something that can pass for "autonomous" in a five-minute walkthrough. It answers questions, it calls an API, it produces a plausible-looking output. None of that requires the model to actually plan, verify its own work, or operate safely without someone watching every step.

The trouble starts when that demo gets scaled into production without anyone checking whether the underlying capability was ever really there. A system that looks impressive answering ten curated questions can fall apart completely on the eleventh, because there was never any planning logic to generalize from — just a good prompt and a lucky run. Multiply that by real budgets, real customer data, and real deadlines, and the gap between "looked agentic" and "was agentic" becomes an expensive one to discover after the fact.

The PROOF Framework

Planning. A real agent decomposes a goal into steps before it acts — it doesn't just react to the next input in a fixed sequence. Ask: show me a case where it chose a different approach than the obvious first guess, because the situation called for it. If every run follows the same script regardless of context, there's no planning happening — just execution of something a human already decided.

Reasoning. A real agent can justify a decision in a way that's checkable against something outside its own output — a log, a source document, a test result — not just a confident-sounding explanation. Ask: make it prove a specific claim using evidence, not memory. Watch what happens when it can't find that evidence. A system with real reasoning will say so. One without it will often just make something up that sounds right.

Orchestration. This is about whether there's a real structure coordinating multiple steps — state passed deliberately between stages, defined roles, handled failure — or whether it's one prompt calling another prompt because someone strung two API calls together and called it a pipeline. Ask: what happens to the rest of the process if one step partially fails? A vague answer, or "it just continues," is a sign there's no real orchestration layer underneath the marketing.

Observability. Can someone actually see what the system is doing while it's doing it — which tools it called, what each step cost, what it considered and rejected — or does it hand you a finished result with no visibility into how it got there? Ask to see the dashboard, not the demo. A system with nothing to show here isn't necessarily broken, but it is a black box, and black boxes are exactly where expensive mistakes hide until they've already happened.

Feedback. Does the system have a real loop that catches its own mistakes — automated evaluations, tests, structured human checkpoints — or does "verification" mean someone skimmed the output and it seemed fine? Ask directly: what caught the last time this was wrong, and how? If nobody has a specific answer to that question, there's no feedback loop, just hope.

Scoring It

Score each of the five categories from 0 to 5 based on specific, checkable evidence — not on how confident the answer sounded. Twenty-five points total.

Score	What You're Actually Looking At
20–25	A genuinely agentic system — autonomy, planning, and safety are real and demonstrable
10–19	Real capability, but under-supervised — it works, and that's exactly what makes it risky if left unchecked
0–9	Not actually agentic — a well-dressed prompt, regardless of what it's called

The scoring isn't the point, though — the conversation it forces is. Whoever is presenting the "agent" — a teammate, a thread, a demo — if they can walk through all five categories with specifics, examples, and a straight answer to "what caught it being wrong," it's almost certainly real. If they get vague, pivot to a different demo, or answer with "the model is really good" instead of a specific mechanism, that's told you everything you need to know without saying it directly.

What to Actually Ask in the Room

You don't need all five questions to get a useful answer. In practice, two do most of the work:

"What's this cost per run, and who's watching that number?" If nobody can answer immediately, there's no real observability, and that alone should worry you more than any technical detail about the model.

"What caught the last time it was wrong?" This is the single best filter in the whole framework. A real feedback loop has a specific, mechanical answer — a test that failed, an eval that flagged it, a human checkpoint that caught it. A hand-wave here means the system has never actually been checked, which is a very different claim than "it works."

If you only remember one line from this, make it the one that started it: cool deck — where's the PROOF?

Building Local AI Agents in Java with Tools4AI and Ollama: An Insurance Claims Use Case

vishalmysore — Tue, 28 Jul 2026 21:20:10 +0000

Tools4AI is a 100% Java agentic AI framework that turns any annotated Java method into an AI-callable action. Ollama runs open models like Llama 3.1 and Phi-4 locally and exposes an OpenAI-compatible API. Point Tools4AI at http://localhost:11434/v1 and you get a fully offline, on-premise AI agent — no data ever leaves your network. In this tutorial we build an insurance claims triage agent that reads a claimant's free-text incident report, routes it to the right business action, extracts structured data, gates high-value payouts behind a human approval, and records a compliance audit trail.

Who is this for? Java developers, solution architects, and engineering leaders in regulated industries (insurance, banking, healthcare) who want agentic AI without sending sensitive data to a third-party API.

Why local AI agents matter for insurance

Insurance runs on personally identifiable information (PII): names, addresses, policy numbers, medical details, vehicle data, and loss descriptions. Sending that data to a hosted LLM API creates regulatory, contractual, and reputational risk. At the same time, claims teams are drowning in unstructured text — First Notice of Loss (FNOL) reports, adjuster notes, emails, and call transcripts.

A local AI agent solves both problems at once:

Data never leaves your premises. The model runs on your own hardware via Ollama.
Deterministic business logic stays in Java. The LLM decides what to do; your audited, tested Java code decides how.
Human-in-the-loop and audit trails are first-class, so you can satisfy compliance reviewers.

That combination — private inference plus governed execution — is exactly what Tools4AI + Ollama gives you.

What is Tools4AI?

Tools4AI (io.github.vishalmysore:tools4ai on Maven Central) is a lightweight, pure-Java agentic AI framework and ADK. Its core idea is simple and powerful:

Annotate a Java class with @Agent and its methods with @Action. Tools4AI scans the classpath, and at runtime it maps a natural-language prompt to the correct method, extracts the parameters, and invokes it — no manual function schemas required.

Key capabilities used in this article:

Feature	What it does
Action routing	Maps a prompt to the right `@Action` method automatically
Automatic parameter mapping	Fills method arguments (including POJOs, lists, maps, dates) from the prompt
POJO transformation	Converts free text into a populated Java object
Risk gating	Blocks `HIGH`-risk actions unless a human approves
Audit trail	Records every action for compliance
Agent memory	Keeps conversation context across turns

Because it is provider-agnostic, the same code runs on Gemini, OpenAI, Anthropic — or, as we will do here, a local Ollama model through its OpenAI-compatible endpoint.

Why Ollama + Java is a great fit for regulated data

Ollama serves open-weight models (Llama 3.1, Phi-4, Mistral, Gemma, and more) behind an OpenAI-compatible REST API at http://localhost:11434/v1. Because Tools4AI's OpenAiActionProcessor already speaks that protocol, wiring the two together is pure configuration — no adapter, no new code.

✅ Zero data egress — inference is local.
✅ No API keys, no per-token billing.
✅ Runs in air-gapped / VPC environments.
✅ Same Tools4AI code works with a cloud provider later if you choose.

Prerequisites

Java 8+ (Tools4AI compiles at Java 8 bytecode) and Maven
Ollama installed — download here
~8 GB RAM free for a capable instruction model (more is better)

Step 1 — Install and run Ollama

Pull a model that is good at function calling and start it:

ollama pull llama3.1
ollama run llama3.1

Ollama now serves the OpenAI-compatible API locally. Verify it:

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"llama3.1","messages":[{"role":"user","content":"Say OK"}],"stream":false}'

You should get a JSON response with "content": "OK".

Step 2 — Add the Tools4AI dependency

<dependency>
    <groupId>io.github.vishalmysore</groupId>
    <artifactId>tools4ai</artifactId>
    <version>1.2.1</version>
</dependency>

Check Maven Central for the latest version. If you build from source, remember to enable the -parameters compiler flag so Tools4AI can read method parameter names.

Step 3 — Point Tools4AI at Ollama

Create tools4ai.properties on your classpath (e.g. src/main/resources/tools4ai.properties):

## Any non-empty value works — Ollama ignores the key,
## but Tools4AI only builds the model when a key is present.
openAiKey=ollama

## Ollama's OpenAI-compatible base URL
openAiBaseURL=http://localhost:11434/v1

## Use the exact Ollama model tag
openAiModelName=llama3.1

Two gotchas worth knowing:

LocalAIActionProcessor is a stub — its methods return null. Do not use it. The OpenAI-compatible route above is the working path.
Properties file wins over -D VM options. Tools4AI reads tools4ai.properties first and only falls back to -DopenAiModelName=... if the file value is empty. If you rely on VM options, leave the corresponding property blank.

Step 4 — Your first local AI call

import com.t4a.processor.OpenAiActionProcessor;

public class Hello {
    public static void main(String[] args) throws Exception {
        OpenAiActionProcessor processor = new OpenAiActionProcessor();
        String reply = processor.query("Reply with exactly one word: PONG");
        System.out.println(reply); // -> PONG
    }
}

That single query() call already round-trips through Tools4AI's config loader, langchain4j, and your local Ollama model. If it prints PONG, you are fully offline and ready to build an agent.

The insurance use case: an FNOL claims triage agent

Let's build something real: an agent that handles the First Notice of Loss (FNOL) — the moment a policyholder reports an incident. Our agent will:

Route a claimant's message to the right business action (check policy, file a claim, estimate a payout, or escalate).
Extract a structured claim record from a free-text incident description.
Gate large payouts behind a mandatory human approval.
Audit every action for compliance.

1. Define your business actions as agents

This is the whole point of Tools4AI: your business logic is just an ordinary Java class. Annotate the class with @Agent, annotate each callable method with @Action, and you're done — no interface to implement, no boilerplate. Tools4AI scans the classpath, registers every @Action method (many per class is fine), instantiates the class for you, and reads riskLevel straight from the annotation.

import com.t4a.annotations.Action;
import com.t4a.annotations.Agent;
import com.t4a.api.ActionRisk;

@Agent(groupName = "claims", groupDescription = "insurance policy and claims actions")
public class ClaimsAgent {

    @Action(description = "check whether an insurance policy is active and in good standing "
            + "given its policy number")
    public String checkPolicyStatus(String policyNumber) {
        return "Policy " + policyNumber + " is ACTIVE | coverage: AUTO | deductible: $500";
    }

    @Action(description = "open a new insurance claim for a policyholder describing an incident, "
            + "given the policy number, the incident type (collision, theft, fire, water damage) "
            + "and a short description")
    public String fileClaim(String policyNumber, String incidentType, String description) {
        String claimId = "CLM-" + Math.abs((policyNumber + incidentType).hashCode() % 100000);
        return "Opened claim " + claimId + " (" + incidentType + ") for policy " + policyNumber;
    }

    @Action(description = "estimate the payout amount in US dollars for a described incident, "
            + "given the incident type and the estimated damage amount in dollars")
    public String estimatePayout(String incidentType, double estimatedDamage) {
        double payout = Math.max(0, estimatedDamage - 500);   // minus deductible
        return "Estimated payout for " + incidentType + ": $" + payout;
    }

    // Settling a claim moves money. Declaring riskLevel = HIGH right on the annotation is enough:
    // Tools4AI refuses to trigger it via auto-prediction, so it can only be invoked explicitly.
    @Action(description = "approve and settle a claim payout to the policyholder, "
            + "given the claim id, the amount in dollars, and who approved it",
            riskLevel = ActionRisk.HIGH)
    public String settleClaim(String claimId, double amount, String approvedBy) {
        return "SETTLED " + claimId + " for $" + amount + " approved by " + approvedBy;
    }
}

Two things to know. (1) riskLevel belongs on @Action, not @Agent (@Agent only carries groupName, groupDescription, and an optional prompt). (2) The class needs a public no-arg constructor — Tools4AI instantiates it for you.

(There is an older style where an action class implements JavaMethodAction. Avoid it unless you need it: that path registers only the first @Action method per class and ignores @Action(riskLevel=…) — you'd have to split every action into its own class and override getActionRisk(). Plain @Agent classes, as above, have neither limitation.)


### 2. Natural-language action routing

Now let the agent decide which method to call from plain English. No `if/else`, no intent parser — Tools4AI does the mapping.

java
import com.t4a.processor.OpenAiActionProcessor;

OpenAiActionProcessor agent = new OpenAiActionProcessor();

// The AI picks checkPolicyStatus and extracts policyNumber = "AUTO-88213"
Object status = agent.processSingleAction(
"Can you tell me if my policy AUTO-88213 is still active?");
System.out.println(status);
// -> Policy AUTO-88213 is ACTIVE, auto coverage, $500 deductible

// The AI picks fileClaim and extracts all three arguments
Object claim = agent.processSingleAction(
"I need to file a claim on policy AUTO-88213. Someone rear-ended my car " +
"in a parking lot and the bumper is cracked.");
System.out.println(claim);
// -> Opened claim CLM-12345 (collision) for policy AUTO-88213


You can also pass the action explicitly (`agent.processSingleAction(prompt, new FileClaimAction())`) when you already know which capability to use — handy for deterministic, single-purpose endpoints.

### 3. Turn free text into a structured claim (POJO extraction)

Claimants describe incidents in messy prose. Use `OpenAIPromptTransformer` to convert that into a clean Java object you can validate and persist. The `@Prompt` annotation adds per-field instructions such as date formatting.

java
import com.t4a.annotations.Prompt;
import java.util.Date;

public class ClaimReport {
public String claimantName;
public String policyNumber;
public String incidentType; // e.g. collision, theft, fire, water damage

@Prompt(describe = "estimated cost of damage in US dollars, number only")
public double  estimatedDamage;

@Prompt(dateFormat = "yyyy-MM-dd", describe = "date the incident occurred")
public Date    incidentDate;

public String  location;

public String toString() {
    return "ClaimReport{name=" + claimantName + ", policy=" + policyNumber +
           ", type=" + incidentType + ", damage=$" + estimatedDamage +
           ", date=" + incidentDate + ", location=" + location + "}";
}

}

java
import com.t4a.transform.OpenAIPromptTransformer;

OpenAIPromptTransformer transformer = new OpenAIPromptTransformer();

String fnol =
"Hi, this is Priya Sharma, policy HOME-55021. On July 12th 2026 a burst pipe " +
"flooded my kitchen in Austin. A plumber estimated about $3,200 in damage.";

ClaimReport report = (ClaimReport) transformer.transformIntoPojo(fnol, ClaimReport.class.getName());
System.out.println(report);
// -> ClaimReport{name=Priya Sharma, policy=HOME-55021, type=water damage,
// damage=$3200.0, date=2026-07-12, location=Austin}


One call turns an unstructured report into a validated, typed record ready for your claims pipeline.

### 4. Gate high-value payouts with human-in-the-loop

Settling a claim moves money — it must **never** be triggered automatically by the model. With the **core** Tools4AI API you get two guarantees, no extra libraries required:

1. **Auto-prediction refuses `HIGH`-risk actions.** If you call `processSingleAction(prompt)` with no explicit action and the best match is `HIGH` risk, it is not executed.
2. **Explicit calls run only if a human approves.** Pass a `HumanInLoop` approver to `processSingleAction(prompt, action, approver, explain)` — the action runs only when the approver returns valid.

Provide an approver by implementing `HumanInLoop`. In production each call would open a ticket, page an adjuster, or invoke your workflow engine and block until a decision returns:

java
import com.t4a.detect.FeedbackLoop;
import com.t4a.detect.HumanInLoop;
import java.util.Map;

public class AdjusterApproval implements HumanInLoop {
private final boolean approve; // wire to a real approval channel
public AdjusterApproval(boolean approve) { this.approve = approve; }

@Override
public FeedbackLoop allow(String prompt, String methodName, Map<String, Object> params) {
    return () -> approve;
}
@Override
public FeedbackLoop allow(String prompt, String methodName, String params) {
    return () -> approve;   // core calls this String overload with the action JSON
}

}


Because `ClaimsAgent` is a plain `@Agent` POJO (not an `AIAction`), grab the registered action from Tools4AI's registry by name to invoke it explicitly:

java
import com.t4a.api.AIAction;
import com.t4a.predict.PredictionLoader;
import com.t4a.processor.LogginggExplainDecision;
import com.t4a.processor.OpenAiActionProcessor;

OpenAiActionProcessor agent = new OpenAiActionProcessor();
AIAction settle = PredictionLoader.getInstance().getAiAction("settleClaim");

// Human declines -> settleClaim never runs; you get "Human verification failed"
agent.processSingleAction("Settle claim CLM-12345 for $8000, approved by Vishal",
settle, new AdjusterApproval(false), new LogginggExplainDecision());

// Human approves -> the settlement executes
Object result = agent.processSingleAction("Settle claim CLM-12345 for $8000, approved by Vishal",
settle, new AdjusterApproval(true), new LogginggExplainDecision());
// -> SETTLED CLM-12345 for $8000.0 approved by Vishal


> Because `riskLevel = HIGH` is declared on the `@Action`, the auto-prediction refusal in step 1 works with no extra code.

### 5. Going further: the agent toolkit

Beyond the core, Tools4AI ships an **agent toolkit** (`com.t4a.agent.*`) of composable decorators over the `AIProcessor` interface — a two-signoff `RiskGatedActionProcessor`, a JSON `AuditedActionProcessor` for compliance, `InMemoryActionMetrics`, retry/rate-limit resilience, `AgentMemory` for multi-turn claims, and multi-agent orchestration. For example, a compliance audit trail is one wrapper:

java
// Requires a Tools4AI build that includes the agent toolkit (com.t4a.agent.*)
AuditTrail trail = new JsonFileAuditTrail("/var/claims/audit.jsonl");
AIProcessor audited = new AuditedActionProcessor(new OpenAiActionProcessor(), trail);
audited.processSingleAction("File a claim on policy AUTO-88213 for a cracked bumper");


> **Availability note:** the agent toolkit is part of newer Tools4AI builds and may not be in every published Maven Central release. Check that `com.t4a.agent.*` is present in your resolved jar before importing it. The InsureJAI demo above uses **core only**, so it builds against the published artifact as-is. For multi-turn claims, `AgentMemory` / `PersistentFileAgentMemory` from the same toolkit keep conversation context across turns and restarts.

---

## Choosing the right local model

**Model capability directly determines how well function calling works.** In our testing:

- **Capable instruction models** (`llama3.1`, `phi4`) reliably route prompts to the correct `@Action` **and** fill in the arguments — including numbers, dates, and nested POJOs.
- **Very small models** (e.g. `gemma3:270m`) often select the right action but leave parameters empty. They are fine for a plain `query()`, but unreliable for full tool calling.

**Recommendation:** start with `llama3.1` (8B) or `phi4` (14B) for claims-style extraction. Use a larger model if your prompts are long or multi-entity. Match the model to your hardware — bigger models need more RAM/VRAM and are slower on CPU.

## Production hardening: composing the agent stack

Every agent-toolkit capability is a decorator over the `AIProcessor` interface, so they nest in any order — and the order encodes your policy. A production-grade claims agent might look like this (requires the `com.t4a.agent.*` toolkit noted above):

java
AIProcessor claimsAgent =
new AuditedActionProcessor( // records the final outcome
new MeteredActionProcessor( // latency + error rates
new RiskGatedActionProcessor( // human approval for HIGH-risk settlements
new RetryActionProcessor( // survive transient model hiccups
new OpenAiActionProcessor(), // -> your local Ollama model
3, 500, null),
new AdjusterApproval()),
metrics),
new JsonFileAuditTrail("/var/claims/audit.jsonl"));




Reading outside-in: audit the *final* decision, measure *user-visible* latency, require approval *once* (not per retry), and retry transient failures closest to the model. Rearranging the layers changes the semantics — choose deliberately.

## Frequently asked questions

**Does any claim data leave my network?**
No. With Ollama the model runs locally and Tools4AI talks to `http://localhost:11434`. Nothing is sent to a hosted API.

**Do I need an OpenAI API key?**
No. Set `openAiKey=ollama` (any non-empty placeholder). Ollama ignores it; Tools4AI just needs a non-empty value to initialize the client.

**Why does the model pick the right action but leave the fields blank?**
The model is too small for reliable function calling. Switch to `llama3.1` or `phi4`.

**Can I use the same code with a cloud provider later?**
Yes. Tools4AI is provider-agnostic. Swap the processor (or the base URL/model) and your `@Agent` / `@Action` code is unchanged.

**Is this production-ready?**
The building blocks — risk gating, audit, retry, metrics, memory — are designed for production. As always, validate model outputs, keep humans in the loop for money-moving actions, and test against your own data.

## Conclusion

With **Tools4AI + Ollama** you get the best of both worlds: **private, on-premise LLM inference** and **governed, testable Java business logic**. In a few dozen lines we built an insurance FNOL agent that understands natural language, extracts structured claims, protects high-value payouts behind human approval, and logs everything for compliance — all without a single byte of PII leaving the building.

- 🏥 Runnable project for this article: [github.com/vishalmysore/insureJAI](https://github.com/vishalmysore/insureJAI)
- ⭐ Star the framework: [github.com/vishalmysore/Tools4AI](https://github.com/vishalmysore/Tools4AI)
- 📦 Maven Central: [io.github.vishalmysore:tools4ai](https://central.sonatype.com/artifact/io.github.vishalmysore/tools4ai)
- 📚 Architecture deep-dive: [Tools4AI ARCHITECTURE.md](https://github.com/vishalmysore/Tools4AI/blob/main/ARCHITECTURE.md)

*Build private AI agents in Java. Keep your data where it belongs.*

The Real Reason 40% of Agentic AI Projects Are Getting Cancelled (It's Not the Model)

vishalmysore — Wed, 22 Jul 2026 23:26:30 +0000

Gartner says nearly half of agentic AI initiatives will be scrapped before 2027. I've spent the last year building the four pieces of infrastructure that pilots skip — and production punishes.

Every few weeks there's a new model that's supposed to change everything. Grok 4.5 lands with a viral benchmark fight. GPT-5.6 ships days later. Claude gets a new tier. Somewhere a Gemini release slips again. Each launch comes with the same implicit promise: this one is finally smart enough to run your agents in production.

And yet Gartner now expects over 40% of agentic AI projects to be cancelled before 2027 — not for lack of a capable model, but because of escalating costs, unclear business value, and inadequate risk controls. IBM's numbers tell the same story from a different angle: in a Q1 2025 survey of 2,000 CEOs, only 25% of AI initiatives had delivered the ROI they were expected to. McKinsey's own research shows the same shape of gap — 88% of organizations now use AI in at least one business function, but only 7% say it's fully scaled.

I don't think that gap is a model problem. I've built four separate systems this year, each one forced on me by a different way agents fail in the real world, and not one of those failures had anything to do with which LLM was behind the curtain. It was always a missing layer.

Pilots only need one layer to hold. Production needs all four.

A demo succeeds with a clean prompt, a happy path, and an audience that doesn't push back. Production has none of that mercy. Here are the four layers I keep re-discovering the hard way — and the projects I built once I got tired of discovering them.

1. The Spec Layer — why the agent is "almost right"

The first failure mode isn't a bad answer. It's a plausible one. AI coding assistants and agents don't leave gaps empty — they fill them with confident guesses that compile, pass the demo, and quietly do the wrong thing once real data hits them.

That's the problem I built ZeeSpec to solve: a structured specification framework built on the Zachman Framework's dimensions and the simplicity of 5W1H — What, Where, Who, When, Why, How. The rule is blunt: if you can't answer a question about your system, that system is undefined — and undefined space is space the AI will fill on your behalf. Sixty questions, about an hour, and the agent stops inventing because there's nothing left to invent.

→ Live framework

2. The Guardrail Layer — why "it worked in staging" doesn't mean it's safe

Even a well-specified agent will eventually be handed a request it shouldn't fully honor. This is the layer most pilots never test, because pilots don't have real stakes.

I built Harness Engineering after realizing that the part of an agent stack that actually matters isn't the model call — it's everything wrapped around it. The project routes calls across OpenAI, Anthropic, Gemini, and NVIDIA NIM models interchangeably, but the part I actually demo is the guardrail layer: a hepatotoxicity check that blocks a dangerous drug interaction, a fraud rule that routes a suspicious claim straight to a SIU referral, a penicillin-allergy check that stops a bad recommendation before it reaches anyone. It even ships a Mock AI provider that runs the full tool-calling and guardrail sequence with zero network calls — so every block you see in a demo is real logic, not a scripted illusion.

Swap in a smarter model and none of that logic appears on its own. It has to be engineered in.

→ Source

3. The Memory Layer — why the agent keeps making the same mistake

LLMs are stateless. Every new session is total amnesia — the agent doesn't remember the itinerary it planned yesterday, or the mistake it made suggesting a hotel an hour from the airport. Most teams try to paper over this with RAG, which is good at storing documents but was never designed to store experience.

That's what ReasoningBank is for — an AI travel agent I built as an independent demonstration of the ReasoningBank memory pattern, which structures and persists an agent's actual experience across sessions instead of just its source documents. Instead of re-learning the same failure every single run, the agent's mistakes become part of what it checks against next time.

This is the layer that quietly wrecks ROI math: if your agent re-derives the same lesson on every session, you're paying full inference cost to relearn things you've already paid to learn once.

→ Live demo · Source

4. The Coordination Layer — why the failure isn't in any single agent

The last failure mode only shows up once you have more than one agent, and by 2026 almost nobody has just one. The bugs stop living inside an agent and start living in the seams between agents — who's allowed to call whom, which protocol carries which kind of information, what happens when two agents disagree.

I've spent a chunk of this year building out an Agentic Mesh — a network of specialized agents (in one build, four separate servers handling completely different domains) coordinated through the A2A and MCP protocols running side by side rather than picking one. The point isn't that multi-agent systems are hard to build; it's that they're easy to build badly, because most teams treat the protocol layer as an afterthought instead of the thing that determines whether the mesh survives contact with a real workload.

→ Live pattern

Why this explains the 40%

Line up those four layers against Gartner's number and the pattern gets obvious. A demo only has to clear one of them — usually the spec, because that's the one you control by writing a good prompt. Production has to clear all four, simultaneously, under load, with real users who don't behave like a demo script. Most agentic AI projects don't get cancelled because the model got dumber between the pilot and the rollout. They get cancelled because the team never built the other three layers, and the model was never going to build them either.

That's also why the next model release won't move the 40% number much. None of these four layers live inside the weights. They live in the specification you wrote before the first prompt, the guardrails you wrote around the tool calls, the memory store you built to survive the session boundary, and the protocol you chose to let agents actually talk to each other. Swap GPT-5.6 for Grok 4.5 for whatever ships next month, and an agent with none of those four layers will fail in production exactly as fast as it did before.

If you're scoping an agentic AI project right now, the honest audit isn't "which model should we use." It's: can you answer every question about what this system is (spec), what it's allowed to do when it's wrong (guardrails), what it remembers between sessions (memory), and how it talks to the other agents it will inevitably need (coordination)? Wherever that audit comes back empty is exactly where your project will show up in next year's cancellation statistics.

Disclaimer: the stats below are cited from Gartner, IBM, and McKinsey publications, linked at each mention. Everything else — the four-layer framework, the diagnosis of why projects fail, the connection I'm drawing between the two — is my own opinion, based on my own projects, not a finding from any of those firms.

I write about agentic AI architecture, spec-driven development, and the infrastructure layer nobody puts in the keynote. Links to the projects above, and the rest of my work, are on GitHub and Medium.

Serverless AI in a Browser Tab: Java WebAssembly + Local WebGPU LLMs

vishalmysore — Tue, 30 Jun 2026 22:14:03 +0000

A deep technical whitepaper on building a zero-infrastructure RAG architecture where the business logic is Java compiled to WebAssembly and the intelligence is a quantized LLM running on your own GPU

Reference implementation: github.com/vishalmysore/javaWASM · Live demo: vishalmysore.github.io/javaWASM

Abstract

For a decade the default shape of an "AI application" has been fixed: a thin client in the browser, a fat backend on someone else's servers, and a metered API call to a model hosted in a data center you will never see. This paper describes an architecture that inverts that shape completely. The entire system — document parsing, text chunking, vector storage, similarity search, context compression, multi-agent orchestration, and large-language-model inference — runs inside a single browser tab, on the user's own hardware, with no backend, no database service, and no API key.

Two technologies make this possible. WebAssembly (Wasm) lets us compile a real, statically-typed business-logic core written in Java down to a compact, near-native binary that runs in the browser sandbox. WebGPU gives that same tab direct access to the machine's GPU, so a quantized small language model can generate tokens locally. We show how to weld these together with on-device embeddings and an in-browser vector database to produce a Retrieval-Augmented Generation (RAG) pipeline that is private by construction and costs nothing to operate.

This is not a thought experiment. Every claim here is backed by a deployed, open-source reference implementation, and we are deliberately honest about the sharp edges we hit along the way.

Project Disclaimer & Intent

My intention behind exploring this architecture is to push the boundaries of what is possible and challenge the traditional AI application model.

The question I wanted to explore is: can we move more intelligence and computation closer to the user?

Can edge computing evolve beyond servers? Can technologies like WebAssembly, WebGPU, WebRTC, and local AI models create a new era of applications where privacy, performance, and cost are balanced differently?

This project is not about replacing cloud AI. Large-scale models and centralized infrastructure will continue to play a critical role. Instead, it is an exploration of what becomes possible when parts of the AI stack move from the cloud into the browser.

The goal is to experiment with a future where applications can be more private, more resilient, and less dependent on backend infrastructure — while being honest about the current trade-offs around model size, hardware limitations, and browser maturity.

The future may not be cloud versus edge. It may be a smarter balance between both.

1. The problem with the current architecture

The conventional AI stack has four structural taxes:

Cost. Every inference is a billable event. A feature that "summarizes the user's notes" has a unit economics problem the moment it has users.
Privacy. To get an answer, the user's data must leave their device and transit a third party. For legal documents, medical notes, source code, or personal journals, that is often a non-starter.
Latency & availability. A network round-trip sits on the critical path of every interaction, and your app is only as available as the upstream API.
Operational drag. Servers, autoscaling, key rotation, rate-limit handling, vector-DB clusters — infrastructure that must be built, secured, paid for, and kept alive.

The interesting observation in 2026 is that none of these taxes are fundamental any more. Browsers have quietly become capable of running compute-heavy code at near-native speed (Wasm) and of driving the GPU directly (WebGPU). Embedding models have shrunk to tens of megabytes; instruction-tuned LLMs now ship in sub-gigabyte quantized form. The pieces to move the entire stack into the client exist — they just have not been assembled into a coherent architecture. That assembly is the subject of this paper.

2. What is WebAssembly?

WebAssembly is a portable binary instruction format for a stack-based virtual machine. It is not a language you write; it is a compilation target. C, C++, Rust, Go, Kotlin, and — crucially for us — Java can all be compiled into a .wasm module that any modern browser (≈96% of global sessions) can load and execute.

Three properties matter:

Near-native performance. Wasm is designed to be decoded and JIT/AOT-compiled extremely fast, with a predictable instruction set close to real CPU semantics. For compute-bound work it is typically 1.5×–20× faster than equivalent JavaScript, with far less variance from garbage-collection pauses and de-optimization.
A capability-secure sandbox. A Wasm module has no ambient access to the DOM, the network, the filesystem, or memory outside its own linear heap. Everything it can touch must be explicitly imported from the host. This is a security model, not an afterthought — it is why running untrusted compiled code in a browser is safe.
Language independence. Wasm breaks the JavaScript monoculture of the web. You can bring a mature, statically-typed, heavily-tested codebase in another language to the front end without a rewrite.

2.1 WasmGC: the part that makes Java practical

Early Wasm had a flat linear memory and no notion of managed objects. A language like Java — built around a garbage-collected object heap — had to ship its own GC and memory manager compiled into the module. That worked, but it bloated binaries and fought the host.

WasmGC (the WebAssembly Garbage Collection proposal, now broadly shipped) adds first-class managed heap types to the VM itself. Managed languages can emit struct and array types that the browser's own garbage collector manages. The payoff is dramatic: a Java program compiles to a lean module (our entire business core is a few hundred kilobytes) that shares the host GC, interoperates cleanly with JavaScript objects, and starts fast. WasmGC is the enabling technology that turns "Java in the browser" from a curiosity into an engineering choice.

3. Why Wasm is a game changer

It is tempting to frame Wasm as merely "faster JavaScript." That undersells it. The shift is architectural:

Dimension	Before (JS-only)	With Wasm
Languages	JavaScript / TypeScript	Any language that compiles to Wasm
Performance	JIT, GC-pause-prone	Near-native, predictable
Code reuse	Rewrite backend logic in JS	Compile existing Java/Rust/C++ as-is
Trust boundary	Same-origin scripts	Capability-secure sandbox by default
Where it runs	Browser	Browser, edge, serverless, embedded, plugins

The deepest consequence is that Wasm moves the unit of deployment from "a service" to "a portable binary." The same compiled core can run in a browser tab, on an edge node, inside a serverless function, or embedded in another application — unchanged. For our purposes, the relevant instance of that generality is simple and radical: the business logic that used to require a server can now ship to, and run on, the client.

4. Writing the business core in Java

We chose Java for the core because it is exactly the kind of language Wasm was supposed to liberate: statically typed, with a vast standard library, decades of tooling, and an enormous corpus of existing business logic. The compiler is TeaVM (v0.15), which takes Java bytecode and emits a WasmGC module.

4.1 Separation of concerns

The architecture draws one hard line:

┌──────────────────────────────────────────────┐
│  JAVA WASM CORE  (the "brain" — TeaVM/WasmGC)  │
│  • document chunking                           │
│  • vector storage + cosine similarity          │
│  • top-K retrieval + context assembly          │
│  • context compression (PCA, k-means, etc.)    │
│  • multi-agent supervisor logic                │
│  • UI construction (DOM + canvas)              │
└──────────────────────────────────────────────┘
                 ▲   │   @JSExport / @JSBody
                 │   ▼
┌──────────────────────────────────────────────┐
│  JS HARDWARE LAYER  (the "muscles")            │
│  • Transformers.js — text embeddings           │
│  • WebLLM — LLM inference on WebGPU             │
│  • sqlite-vec (WASM) — vector KNN engine        │
│  • IndexedDB — durable storage                 │
└──────────────────────────────────────────────┘

The Java core owns the deterministic, algorithmic, business logic. The JavaScript layer owns the asynchronous, hardware-bound, I/O work — the things browsers are natively good at. Neither layer reaches into the other's domain; they communicate only across a narrow, explicit boundary.

4.2 The interop surface

TeaVM's JavaScript Object (JSO) layer provides two annotations that constitute the entire bridge:

@JSExport — expose a Java method so JavaScript can call it. After the module loads, exported methods appear on instance.exports:

public class RAGOrchestrator {
    @JSExport
    public static String buildContext(String queryText, String queryCsv, int compress) {
        float[] qv = parseVector(queryCsv);
        String retrieved = db.searchTopContext(qv, 3);          // Java cosine retrieval
        if (compress != 0) {
            return ContextCompressor.compress(queryText, retrieved, 600).text;
        }
        return retrieved;
    }
}

@JSBody — call a JavaScript function from Java by inlining a snippet:

public class NativeAIBridge {
    @JSBody(params = {"systemPrompt", "userQuery", "context"},
            script = "window.streamSLMInference(systemPrompt, userQuery, context);")
    public static native void executeSLM(String systemPrompt, String userQuery, String context);
}

On the JavaScript side, loading the module is three lines:

const teavm = await TeaVM.wasmGC.load("wasm-gc/classes.wasm");
teavm.exports.main([]);
const context = teavm.exports.buildContext(query, csv, 1); // calls into Java/Wasm

4.3 A hard-won lesson: the synchronous boundary

The single most important design constraint is this: you cannot block on a JavaScript Promise from inside synchronous Wasm code. Embedding a sentence (Transformers.js) and generating a token (WebLLM) are inherently asynchronous. If Java calls an async JS function expecting a return value, it gets a Promise, not the data.

The resolution is a clean rule that shaped the whole system: JavaScript owns the async orchestration; Java owns the synchronous compute. When Java needs many embeddings, it does not pull them. Instead it pushes work outward — it splits text and emits each fragment to JS via @JSBody; JS embeds asynchronously and calls back into a Java @JSExport with the finished vector. Across the boundary we pass only String and int/double (vectors travel as comma-separated strings), because those primitive types marshal reliably on WasmGC. This pattern — Java plans, JavaScript awaits, Java computes — recurs in indexing, semantic compression, and agent orchestration alike.

5. The serverless architecture

"Serverless" here is meant literally — not "someone else's servers (FaaS)," but no servers at all.

The build pipeline compiles RAGOrchestrator.java and friends to classes.wasm + a runtime loader, and a GitHub Actions workflow publishes the static bundle (index.html, app.js, the Wasm artifacts) straight to a CDN (GitHub Pages) with no branch tracking. There is no origin server in the request path. The "deployment" is a handful of static files behind a CDN edge.

Everything that a traditional stack would put on the server now lives in the tab:

Traditional tier	In this architecture
API gateway / app server	Java core compiled to Wasm
Vector database (Pinecone, etc.)	sqlite-vec (WASM) + a Java cosine store
Embedding service	Transformers.js on the local CPU
LLM API (OpenAI, etc.)	WebLLM on the local GPU
Durable storage (Postgres)	IndexedDB / OPFS
CDN	CDN (the only thing left)

The economic and privacy consequences fall straight out of the diagram: marginal cost per inference is zero, and no user data ever leaves the device.

6. Local intelligence: WebGPU, on-device embeddings, and a browser LLM

The "intelligence" tier is where 2026's browser capabilities earn their keep.

6.1 WebGPU

WebGPU is the modern successor to WebGL — a low-level API that exposes the GPU for both rendering and general-purpose compute shaders. It is what makes practical LLM inference in a tab possible: the matrix multiplications at the heart of a transformer map onto GPU compute kernels.

6.2 WebLLM + a quantized SLM

We run Qwen2.5-0.5B-Instruct quantized to q4f16_1 (4-bit weights, ~945 MB VRAM) via WebLLM (MLC). The model weights download once into the browser cache; thereafter inference is fully local and offline-capable. Tokens stream from the GPU directly into the DOM:

const engine = await webllm.CreateMLCEngine("Qwen2.5-0.5B-Instruct-q4f16_1-MLC", { initProgressCallback });
const stream = await engine.chat.completions.create({ messages, stream: true });
for await (const chunk of stream) outputBox.innerHTML += chunk.choices[0]?.delta?.content || "";

6.3 On-device embeddings

Retrieval needs vectors. Transformers.js runs Xenova/all-MiniLM-L6-v2 (384-dimensional, mean-pooled, L2-normalized) on the CPU via ONNX-runtime-web — a ~25 MB model that produces sentence embeddings in milliseconds, entirely client-side.

The division of labor is deliberate: embeddings on the CPU (small, frequent, latency-sensitive), generation on the GPU (large, occasional, throughput-bound), everything else in Wasm.

7. The combined architecture: serverless RAG

Putting the tiers together yields a full Retrieval-Augmented Generation loop with no server in sight.

[User uploads text / PDF]
        │
        ▼
┌───────────────────────────────┐
│  JAVA WASM CORE                │  1. chunk the document (sliding window)
└───────┬───────────────────────┘
        │  emit each chunk ──► JS
        ▼
┌───────────────────────────────┐
│  JS: Transformers.js           │  2. embed each chunk → float[384]
└───────┬───────────────────────┘
        │  vector (CSV) ──► back into Wasm
        ▼
┌───────────────────────────────┐
│  JAVA WASM CORE                │  3. index the vector
│                                │  4. on query: cosine rank + top-K
│                                │  5. compress context (Headroom-style)
└───────┬───────────────────────┘
        │  assembled prompt ──► JS
        ▼
┌───────────────────────────────┐
│  JS: WebLLM (WebGPU)           │  6. stream the answer locally
└───────────────────────────────┘

On top of this spine the reference implementation layers several capabilities, each chosen to demonstrate that real work — not glue — runs in Java/Wasm:

Persistent memory. A "remember/recall" facility backed by sqlite-vec (a vector-search SQLite extension compiled to WASM) for KNN, with IndexedDB as the durable layer; the store is rehydrated into the engine on every boot, so memory survives closing the browser. The Java core orchestrates it; a brute-force JS cosine store stands by as a fallback so the feature degrades gracefully if the WASM engine fails to load.
Context compression ("Headroom-style"). Before the prompt reaches the model, Java trims retrieved context to the query-relevant sentences, drops near-duplicates (Jaccard), and enforces a token budget — in two modes: lexical (term-overlap scoring) and semantic (Java cosine-scores each sentence against the query embedding). Less noise for a small model; fewer tokens for a tight context window.
A multi-agent society. A Supervisor implemented in Java plans a pipeline and authors the role prompts for Researcher → Coder → Critic agents; JavaScript runs the asynchronous LLM turns; Java merges the outputs. The orchestration logic is Wasm; the inference is WebGPU.
A semantic map. Java implements PCA (top-2 components via power iteration, never materializing the 384×384 covariance) and k-means, from scratch, on the chunk embeddings, then draws an interactive, topic-clustered 2D scatter on an HTML <canvas> — real machine learning and graphics, both inside the Wasm core.
A UI built in WebAssembly. A live dashboard whose DOM is created, styled, and event-wired entirely from Java via TeaVM's JSO DOM API — proof that Wasm can own presentation, not just computation.

7.1 Why the Java core is genuine business logic, not a wrapper

A fair skeptic asks: is Java actually doing anything, or just relaying? The answer is concrete. The cosine similarity, the top-K ranking, the sliding-window chunker, the sentence-level compression scoring, the PCA eigen-decomposition, the k-means clustering, and the agent merge are all pure Java numerical/algorithmic code executing in the .wasm. The system even surfaces this: a "Wasm Core Activity" panel streams lines emitted from inside the module (e.g. cosine scores that exist nowhere else), and a selfTest() export computes a known value to prove the compiled code is live.

8. Analysis

8.1 Privacy

The strongest property is structural, not promised: data cannot leak because it never has anywhere to go. Documents, embeddings, memories, and prompts live in the tab's heap, IndexedDB, and GPU memory. The only network traffic is the one-time, cacheable download of static assets and model weights from a CDN. This is a qualitatively different privacy posture than "we don't train on your data."

8.2 Cost

Capital and marginal cost of inference is zero to the operator; compute is donated by the client's own hardware. Hosting is a static CDN bucket. This changes which products are viable — features whose per-call economics would sink a server-backed app are free here.

8.3 Performance & honest trade-offs

We are deliberately not selling a miracle:

Model capability. A 0.5B-parameter model is not GPT-class. It is excellent for grounded, context-bounded RAG answers and demonstrations; it is not a general reasoner. The architecture is model-agnostic — swap in a larger quantized model as client GPUs allow.
Cold start. The first visit downloads model weights (hundreds of MB). Subsequent loads are cache-fast and offline-capable, but the first run is heavy.
Hardware floor. WebGPU requires a reasonably modern browser/GPU (Chrome/Edge 113+). Where WebGPU is absent, retrieval and embeddings still work; generation is disabled.
Lexical vs semantic compression. Term-overlap compression is fast but can drop a relevant sentence on vocabulary mismatch; the semantic mode fixes that at the cost of N extra embedding calls.
Ecosystem maturity. The bleeding edge bites. We hit a real Emscripten 4.0.0 postRun regression in one sqlite-vec WASM build and had to move to a build carrying the upstream fix — and we keep a JS fallback precisely because browser-WASM library stability is still uneven.

These are engineering constraints to plan around, not refutations of the thesis.

9. When to reach for this architecture

This pattern is a strong fit when:

privacy is paramount (legal, medical, financial, personal, on-device enterprise data);
per-inference cost must be zero or near-zero at scale;
offline or air-gapped operation is valuable;
you want to reuse a mature, typed business-logic codebase (Java/Kotlin/Rust) rather than rewrite it in JS;
the task is bounded and retrieval-grounded rather than open-ended frontier reasoning.

It is a poor fit when you genuinely need frontier-model capability, must support thin/old clients with no WebGPU, or require centralized data aggregation.

10. Future directions

Larger local models as quantization and client GPUs improve (1.5B–3B class in the same architecture).
OPFS-backed persistence to drop the IndexedDB rehydration step and keep a true SQLite file on disk.
WebGPU compute from Wasm directly, letting the Java core dispatch its own kernels for the heavy linear algebra.
Embedding caches so semantic compression and the semantic map avoid recomputation.
A portable core — the same Wasm business module redeployed unchanged to edge functions and native hosts, fulfilling Wasm's "write once, run at any tier" promise.

11. Conclusion

The center of gravity of software is shifting back toward the client — not because the cloud failed, but because the browser quietly became a capable, GPU-accelerated, multi-language runtime. WebAssembly lets us put a real, statically-typed business core — written in Java — on that runtime as a lean, sandboxed, near-native binary. WebGPU puts a language model on the same tab's GPU. Stitched together with on-device embeddings and an in-browser vector database, the result is an AI application that is private by construction, free to operate, and dependent on no server at all.

The reference implementation proves it is buildable today, sharp edges and all. The interesting question is no longer whether the full AI stack can live in a browser tab — it can — but which applications should. For anything where privacy, cost, or offline operation matters, the answer is increasingly: this one.

Appendix: reference stack

Layer	Technology	Role
Business core	Java 17 → TeaVM 0.15 (WasmGC)	Chunking, vectors, retrieval, compression, agents, UI
LLM inference	WebLLM (MLC) + `Qwen2.5-0.5B-Instruct-q4f16_1`	Local generation on WebGPU
Embeddings	Transformers.js + `Xenova/all-MiniLM-L6-v2` (384-d)	On-device CPU vectors
Vector engine	sqlite-vec (WASM) `vec0` KNN + Java cosine fallback	Similarity search
Durable storage	IndexedDB	Persistent memory
UI / graphics	TeaVM JSO DOM + Canvas	DOM and the semantic-map canvas, built in Java
Delivery	GitHub Actions → GitHub Pages (CDN)	Static, serverless deployment

Repository: github.com/vishalmysore/javaWASM — Apache-2.0.

This whitepaper documents a working system; its architecture, code snippets, and trade-offs are taken directly from the reference implementation.

webSLM: Fine-tuning, Compiling, and Running Domain-Specific Small Language Models Entirely in the Browser

vishalmysore — Mon, 29 Jun 2026 13:42:20 +0000

webSLM is an end-to-end pipeline for turning a general-purpose Small Language Model (SLM) into a domain-specialized assistant that runs 100% in the browser — no server, no API key, no inference cost, full offline capability after first load. This paper documents the complete lifecycle of a worked example, WebSLM-Medical-0.5B: (1) LoRA fine-tuning of Qwen2.5-0.5B-Instruct on a small domain dataset using a free Colab T4; (2) compilation and 4-bit quantization to a WebGPU model library via a reproducible, GPU-free GitHub Actions workflow built on MLC-LLM v0.19.0; and (3) in-browser execution through WebLLM, including the non-obvious runtime-integration details that determine whether the model loads at all.

We also report a controlled A/B validation showing what fine-tuning on a small dataset actually changes — and what it does not — and document two decoding pitfalls (greedy repetition loops and penalty-induced language drift on bilingual base models) that materially affect output quality on sub-1B models.

1. Introduction

The dominant narrative in language modeling has been scale. But a parallel track — Small Language Models in the 0.5B–7B range, designed to be efficient rather than pruned down — has matured to the point where a properly fine-tuned 0.5B–1.5B model delivers genuinely useful behavior in a focused domain. At the same time, WebGPU has made it possible to run these models directly inside a browser tab, on the user's own hardware.

The gap webSLM closes is not a research problem; it is an engineering and tooling problem. Getting from a Hugging Face checkpoint to a working in-browser, domain-specialized chatbot requires stitching together three independently fiddly stages — fine-tuning, MLC compilation/quantization, and WebLLM runtime wiring — each with version-sensitive environments and undocumented failure modes. This paper is the field manual for that path.

1.1 Design principles

Model-agnostic. Any MLC-LLM-supported base works (Qwen2/2.5, Llama-3.x, Gemma-2, Phi-3.5, Mistral). Nothing in the pipeline is specific to one architecture.
Training and compilation are separate steps, on purpose. Fine-tuning needs a GPU (Colab); compilation is CPU-only codegen (CI). The two never run in the same environment, so neither inherits the other's dependency constraints.
Reproducible, GPU-free builds. The expensive, error-prone compilation runs in GitHub Actions with a pinned, from-source toolchain — no local Linux GPU box required.
Browser-first deployment. The artifact is static files (weight shards + a .wasm) on a CDN; the client is one HTML page.

2. Background

2.1 SLM vs. quantized LLM

These are frequently conflated but are different answers to the same "large models are expensive" problem. An SLM is small by design — its efficiency comes from architecture and curated training data. A quantized LLM is a large model compressed after training; it keeps the full parameter structure and capability profile of the original, just at lower numeric precision. For the browser, only the SLM path is viable: a 1.5B model uses ~1–2 GB of GPU memory, while an INT4-quantized 70B model still needs 30–40 GB. They are not in the same deployment category.

Note that webSLM uses quantization on top of an SLM: the 0.5B model is itself quantized to 4-bit (q4f16_1) for the browser. Quantization here is a deployment compression, not the source of "smallness."

2.2 Fine-tuned SLM vs. RAG

RAG (Retrieval-Augmented Generation) is a system architecture, not a model type: it injects retrieved documents into the prompt at query time. It excels at large, dynamic knowledge bases but requires a retrieval layer (embeddings, vector store, ingestion) — infrastructure that does not exist in a pure browser deployment. Fine-tuning encodes behavior, style, and domain patterns into the weights instead. The two are complementary; for the serverless browser case, fine-tuning is the only specialization mechanism available. (The companion demo includes an optional in-browser TF-IDF RAG path for comparison, but the model itself carries no retrieval.)

2.3 The runtime stack: WebLLM + MLC-LLM

WebLLM (github.com/mlc-ai/web-llm) is the browser runtime. It executes models on WebGPU and exposes an OpenAI-compatible JS API (engine.chat.completions.create()), with streaming, fully local. It cannot load a raw Hugging Face checkpoint.
MLC-LLM (part of the Apache TVM ecosystem) is the compiler. For a browser target it produces two things WebLLM consumes:
1. Quantized weight shards (params_shard_*.bin) plus a manifest (ndarray-cache.json) and chat/sampling config (mlc-chat-config.json).
2. A .wasm model library containing the compiled compute kernels for that specific architecture.

The critical, easily-missed consequence: the .wasm is tied to the runtime ABI. A model library compiled with MLC-LLM v0.19.0 must be loaded by the matching @mlc-ai/web-llm build (0.2.79). Mismatched versions fail to instantiate the wasm.

3. System architecture

  domain data (JSONL chat)
        │
        │   STAGE 1 — Colab T4 (GPU)
        │   finetune/train_lora.py:  LoRA SFT → merge_and_unload → push
        ▼
  merged HF checkpoint                       e.g. VishalMysore/WebSLM-Medical-0.5B
  (standard fp16 safetensors)                (qwen2, 0.5B, full weights)
        │
        │   STAGE 2 — GitHub Actions (CPU only)
        │   normalize_config → convert_weight (q4f16_1) → gen_config → compile --device webgpu
        ▼
  MLC artifacts                              VishalMysore/WebSLM-Custom-MLC
  (params_shard_*.bin + .wasm + configs)     (8 shards ~278 MB + 3.7 MB wasm)
        │
        │   STAGE 3 — Any WebGPU browser (client GPU)
        │   @mlc-ai/web-llm@0.2.79: MLCEngine(appConfig) → reload → chat.completions
        ▼
  in-browser domain assistant               webSLMDemo (GitHub Pages)

Three environments, three hardware profiles, zero shared dependencies. The handoff between stages is always a plain artifact (an HF repo), never a live process.

4. Stage 1 — Fine-tuning on Colab (LoRA SFT)

4.1 Data format

Training data is chat/conversational JSONL — one JSON object per line:

{"messages":[
  {"role":"system","content":"You are a careful medical information assistant. Provide general, educational health information in plain language, and always recommend consulting a licensed healthcare professional for diagnosis or treatment."},
  {"role":"user","content":"What are common signs of dehydration?"},
  {"role":"assistant","content":"Common signs include increased thirst, dry mouth, dark yellow urine, reduced urination, fatigue, dizziness, and headache. Severe dehydration can cause confusion or a rapid heartbeat and needs urgent care. Drink fluids and seek medical help if symptoms are severe or persist."}
]}

Two design rules carry most of the signal:

Keep the system message identical across the dataset. The model learns a stable persona from it. (This system prompt becomes the one you should use at inference — see §6.4 and §7.)
The assistant turns are exactly what the model imitates. Their length, register, and structure are the behavior you are training. The worked example's answers are short, plain-language, and always close with a referral to a professional — and that is precisely the signature the fine-tune learns.

The repository ships three illustrative starter sets in finetune/data/ (medical.jsonl ≈ 34 examples, plus legal and insurance). These are deliberately tiny: they prove the pipeline, they do not fully specialize a model. Real domain quality needs hundreds to thousands of examples.

4.2 The training recipe

Fine-tuning uses LoRA (Low-Rank Adaptation) via TRL's SFTTrainer, rendering each example through the base model's chat template before training. The merged result is a standard HF checkpoint the build stage can consume directly.

Hyperparameter	Value	Notes
Base model	`Qwen/Qwen2.5-0.5B-Instruct`	any Instruct SLM; `arch=qwen2`, `conv=qwen2` flow straight to build
LoRA rank `r`	16
LoRA `alpha`	32
LoRA dropout	0.05
Target modules	`q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj`	all attention + MLP projections
Epochs	3
Learning rate	2e-4
Per-device batch	2
Grad accumulation	8	effective batch ≈ 16
Max sequence length	1024
Precision	fp16	QLoRA (`--bits 4`, bitsandbytes) available for larger bases on 16 GB GPUs
Packing	off

The core of train_lora.py:

tok = AutoTokenizer.from_pretrained(args.base, trust_remote_code=True)
ds  = load_dataset("json", data_files=args.data, split="train")
ds  = ds.map(lambda ex: {"text": tok.apply_chat_template(ex["messages"], tokenize=False)},
             remove_columns=ds.column_names)

lora = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05, bias="none",
                  task_type="CAUSAL_LM",
                  target_modules=["q_proj","k_proj","v_proj","o_proj",
                                  "gate_proj","up_proj","down_proj"])

trainer = SFTTrainer(model=model, train_dataset=ds, args=sft_cfg,
                     peft_config=lora, processing_class=tok)   # falls back to tokenizer= on older TRL
trainer.train()

After training, the adapter is merged back into the base in fp16 and saved as a self-contained checkpoint (this is mandatory — MLC-LLM's convert_weight expects a fused model, not LoRA deltas):

base   = AutoModelForCausalLM.from_pretrained(args.base, torch_dtype=torch.float16)
merged = PeftModel.from_pretrained(base, adapter_dir).merge_and_unload()
merged.save_pretrained(out, safe_serialization=True)
tok.save_pretrained(out)               # carry the tokenizer so the dir is self-contained
merged.push_to_hub("yourname/WebSLM-Medical-0.5B")

If you trained a LoRA adapter elsewhere (axolotl, Unsloth, raw PEFT), merge_lora.py does just the merge+export step. If you did a full fine-tune (no LoRA) you already have a standard checkpoint — skip merging entirely.

4.3 Running it on Colab

The fastest path is the clone-and-run notebook finetune/finetune_webslm_colab.ipynb (Runtime → T4 GPU). It clones the repo, installs finetune/requirements.txt, logs into Hugging Face, trains on the chosen domain, and pushes the merged model to your HF account. On a T4, a few hundred examples train in well under an hour. The final cell prints exactly what to enter in the build Action.

Equivalent CLI:

pip install -r finetune/requirements.txt
python finetune/train_lora.py \
    --base Qwen/Qwen2.5-0.5B-Instruct \
    --data finetune/data/medical.jsonl \
    --push-merged yourname/WebSLM-Medical-0.5B \
    --epochs 3

The output of Stage 1 is a plain fp16 HF model repo — e.g. VishalMysore/WebSLM-Medical-0.5B (qwen2 architecture, single model.safetensors, tokenizer, configs). Nothing browser-specific yet.

5. Stage 2 — Compiling to WebGPU via GitHub Actions

This is the stage that, done by hand, costs newcomers a day or more. webSLM encodes the entire toolchain build and the three-command MLC pipeline into .github/workflows/build-slm.yml, triggered manually (workflow_dispatch) with a domain preset or a Custom path pointing at your fine-tune.

5.1 Why build the toolchain from source

The workflow builds MLC-LLM v0.19.0 and TVM from source, because the MLC nightly wheels are broken by the in-progress apache-tvm-ffi migration. The compile step is CPU-only: mlc_llm compile --device webgpu is code generation via emscripten — it emits WebAssembly kernels and never needs a GPU. A full toolchain build is ~35–45 minutes; the workflow has a 350-minute ceiling and frees disk space first.

Build environment (the parts that matter):

Component	Pin / setting	Reason
MLC-LLM	v0.19.0, from source	nightly wheels broken (tvm-ffi migration)
TVM	bundled `3rdparty/tvm`, from source	`USE_LLVM "llvm-config --link-static"`, `HIDE_PRIVATE_SYMBOLS ON`, all GPU backends OFF
LLVM	system `llvm-dev` + matching `libpolly-*-dev`	TVM static-links LLVM incl. Polly; `llvm-dev` doesn't ship `libPolly.a`
emscripten	3.1.56	wasm toolchain matching MLC-LLM's web runtime
Rust	latest stable (≥1.85)	modern deps; `tokenizers-cpp` needs `--cap-lints=allow` on new Rust

A subtle linking detail handled by the workflow: mlc_llm compile --device webgpu links several wasm bitcode runtimes — mlc_wasm_runtime.bc (from mlc-llm/web) and wasm_runtime.bc / tvmjs_support.bc / webgpu_runtime.bc (from tvm/web). The latter three are built and copied into TVM's build/ so its find_lib_path discovers them.

5.2 The MLC compile pipeline

Once the toolchain exists, the actual conversion is three commands (mirrored in build.sh for local/WSL2 runs), preceded by a config-normalization step:

# 1b. Make a freshly-merged (newer-transformers) config readable by mlc-llm v0.19.0
python normalize_config.py "hf/$NAME"

# 2. Quantize + shard the weights  (HF -> MLC params)
mlc_llm convert_weight "hf/$NAME" --quantization q4f16_1 --model-type qwen2 -o "$W"

# 3. Emit chat template + tokenizer + model metadata
mlc_llm gen_config "hf/$NAME" --quantization q4f16_1 --model-type qwen2 \
    --conv-template qwen2 --prefill-chunk-size 1024 -o "$W"

# 4. Codegen the WebGPU model library
mlc_llm compile "$W/mlc-chat-config.json" --device webgpu \
    -o "$L/$NAME-q4f16_1-webgpu.wasm"

5.2.1 The config-normalization gotcha

Recent transformers releases changed the config schema: the RoPE base moved from a top-level rope_theta into a nested rope_parameters/rope_scaling dict, and torch_dtype became dtype. MLC-LLM v0.19.0 still expects the old top-level keys, so convert_weight on a freshly fine-tuned model fails with:

TypeError: QWen2Config.__init__() missing 1 required positional argument: 'rope_theta'

normalize_config.py hoists rope_theta back to the top level and restores torch_dtype — decoupling your training transformers version from the pinned, old build compiler. It is idempotent and safe on already-old configs:

if "rope_theta" not in cfg:
    for key in ("rope_parameters", "rope_scaling"):
        rp = cfg.get(key)
        if isinstance(rp, dict) and rp.get("rope_theta") is not None:
            cfg["rope_theta"] = rp["rope_theta"]; break
if "torch_dtype" not in cfg and "dtype" in cfg:
    cfg["torch_dtype"] = cfg["dtype"]

This is the kind of failure that, undiagnosed, produces a silent crash or garbage output with no upstream documentation.

5.2.2 Quantization choice

Default is q4f16_1 (4-bit weights, fp16 activations) — the smallest practical format. Some models overflow fp16 to NaN during inference (observed with TinyLlama-1.1B); for those, fall back to q4f32_1 (fp32 activations), trading size for numerical stability. The selected format is part of the wasm filename and the model_id, so it must match between the compiled artifact and the browser config.

5.3 Inputs and outputs

The workflow exposes domain presets (Qwen2.5-Coder-1.5B for code, Qwen2.5-Math-1.5B for math, general Qwen/Llama/Gemma/Phi bases) and a Custom path. For a fine-tune you select Custom and pass:

domain          = Custom
custom_model_hf = VishalMysore/WebSLM-Medical-0.5B
custom_arch     = qwen2
custom_conv     = qwen2
quant           = q4f16_1
custom_name     = WebSLM-Custom        (becomes the output repo/lib name)
upload_hf       = true                 (push artifacts to HF; needs HF_TOKEN + HF_NAMESPACE)

Outputs are saved unconditionally (so a bad HF token never loses a 45-minute build): a downloadable Actions artifact, a GitHub Release carrying the .wasm, and — when upload_hf is set — a self-contained Hugging Face model repo. The worked example produced VishalMysore/WebSLM-Custom-MLC:

File	Size	Role
`mlc-chat-config.json`	2 KB	chat template, sampling defaults, context window
`ndarray-cache.json`	102 KB	weight-shard manifest
`params_shard_0…7.bin`	~278 MB total	4-bit quantized weights (8 shards)
`tokenizer.json`, `tokenizer_config.json`	~11 MB	tokenizer
`libs/WebSLM-Custom-q4f16_1-webgpu.wasm`	3.7 MB	WebGPU model library

Compiled metadata of note: model_type: qwen2, quantization: q4f16_1, context_window_size: 32768, vocab_size: 151936, conv_template: qwen2, default sampling temperature: 0.7, top_p: 0.8.

6. Stage 3 — Running in the browser with WebLLM

The client is a static page importing @mlc-ai/web-llm. The model loads from its HF URLs, caches in the browser (Cache API / IndexedDB), and runs on the client GPU. Three integration details determine whether it works at all — each one produced a distinct, opaque error during bring-up.

6.1 Registering a custom model — `appConfig` goes in the constructor

WebLLM only knows its built-in (prebuilt) models unless you give it an appConfig describing yours. The appConfig must be passed to the MLCEngine constructor, not to reload():

import * as webllm from "https://esm.run/@mlc-ai/web-llm@0.2.79";

const appConfig = {
  model_list: [{
    model:     "https://huggingface.co/VishalMysore/WebSLM-Custom-MLC",          // FULL HF URL
    model_id:  "WebSLM-Custom-q4f16_1-webgpu",                                   // arbitrary local name
    model_lib: "https://huggingface.co/VishalMysore/WebSLM-Custom-MLC/resolve/main/libs/WebSLM-Custom-q4f16_1-webgpu.wasm",
  }],
};

const engine = new webllm.MLCEngine({ appConfig, initProgressCallback });
await engine.reload("WebSLM-Custom-q4f16_1-webgpu");

reload(modelId, chatOpts?)'s second argument is ChatOptions, which has no appConfig field — passing appConfig there is silently dropped, the engine falls back to its prebuilt list, and you get:

Cannot find model record in appConfig for WebSLM-Custom-q4f16_1-webgpu.

6.2 `model` must be a full URL

The model_list[].model field must be a complete https://huggingface.co/{USER}/{REPO} URL (the four accepted forms all start with https://). A bare repo id makes WebLLM's internal new URL(...) throw:

Failed to construct 'URL': Invalid URL

model_lib is the full /resolve/main/.../*.wasm URL; model_id is a free-form local handle.

6.3 Pin the runtime to the wasm's ABI

The import must be version-pinned to the web-llm build matching the MLC-LLM that produced the wasm:

import * as webllm from "https://esm.run/@mlc-ai/web-llm@0.2.79";   // NOT unpinned (=latest)

esm.run/@mlc-ai/web-llm with no version resolves to latest, whose wasm ABI can differ from a v0.19.0-compiled library — producing instantiation failures at load. The pin table:

Built with	Runtime
MLC-LLM v0.19.0	`@mlc-ai/web-llm@0.2.79`

6.4 Inference and decoding

WebLLM's API is OpenAI-shaped and streams:

const stream = await engine.chat.completions.create({
  messages: [
    { role: "system", content: TRAINING_SYSTEM_PROMPT },   // match the prompt the model was TRAINED with
    { role: "user",   content: query },
  ],
  stream: true,
  temperature: 0.7, top_p: 0.8, max_tokens: 256,            // Qwen2.5's recommended sampling
});
for await (const chunk of stream) { /* chunk.choices[0].delta.content */ }

Two points are decisive for output quality on a 0.5B model (see §7.3):

Use the training system prompt at inference. The fine-tune's behavior is conditioned on the persona it was trained with; a different system prompt pulls it back toward generic base behavior.
Use the model's recommended sampling (temperature 0.7, top_p 0.8). Greedy decoding and aggressive penalties both degrade small-model output in characteristic ways.

6.5 The demo application

The companion demo (webSLMDemo, deployed to GitHub Pages) has two modes:

Product demo — Base + RAG (left) vs. Fine-tuned SLM (right). The base panel runs a general model with an in-browser TF-IDF retriever injecting document context; the SLM panel runs the fine-tune with a domain system prompt and no retrieval. This contrasts the two specialization strategies.
Fine-tuning proof — a controlled A/B (next section). It loads the exact base the fine-tune started from (Qwen2.5-0.5B-Instruct-q4f16_1-MLC, a WebLLM prebuilt at the same quantization) on the left and the fine-tune on the right, feeds both the identical training system prompt with no retrieval, and uses identical decoding — so the only variable is the LoRA training.

7. Validation: what fine-tuning actually changed

Because the proof mode holds base, prompt, and decoding identical across both panels, any difference is attributable to the LoRA fine-tuning. The following are verbatim in-browser outputs.

7.1 In-distribution question (trained topic)

Prompt: "What are common signs of dehydration?" (a question whose topic is in the training set.)

Base — Qwen2.5-0.5B, no fine-tune: "1. Sunken Eyes… 6. Confusion… 9. Dry Skin: Not having much moisture in your skin. 10. Dry Skin… 11. Dry Skin…" — rambling, and falls into a repeat loop to the token limit.

Fine-tune — WebSLM-Medical-0.5B: "Common signs of dehydration include: 1. Sunken eyes 2. Dry mouth and lips … 4. Urine that is dark or less than normal … 7. Not sweating heavily or feeling weak. These symptoms can be caused by low fluid intake, heatstroke… If you notice any of these signs, it's important to seek medical attention right away."

The fine-tune reproduces the trained content (thirst, dark urine, reduced urination, rapid heartbeat — closely mirroring its medical.jsonl answer), is concise, stops cleanly, and closes with a referral — exactly the trained signature. The base is verbose and unstable. Fine-tuning here improved both style and generation stability.

7.2 Held-out question (untrained topic)

Prompt: "What is long covid?" (a topic the 34-example dataset never covered.)

Base: invents an alias ("also known as Long-Term Exposure") but gets the chronic/months-to-years framing roughly right.
Fine-tune: cleaner structure and plausible symptom list, but states symptoms last "an average of two to three weeks" — which is wrong (that is acute COVID; long COVID lasts months by definition).

Honest finding: on a held-out topic the fine-tuning signal is weak and both models hallucinate. A 34-example LoRA imparts style and in-distribution phrasing, not reliable new knowledge, and certainly not factual reliability on topics outside the training distribution. This is expected and is the central caveat for small-data fine-tuning.

7.3 Decoding pitfalls (general to sub-1B WebLLM models)

Decoding choice changes outputs as much as fine-tuning does on these models:

Decoding	Effect on the 0.5B models
Greedy (`temperature: 0`)	Degenerate repetition loops ("Confusion. Confusion…") that bury the trained style
Low temp + strong `frequency_penalty`/`presence_penalty`	Language drift: penalizing repeated English tokens pushes a bilingual Qwen model into Chinese tokens → gibberish loops ("答? 答答?")
`temperature 0.7`, `top_p 0.8` (Qwen's own recommended)	Stable, coherent, no loops, no drift — the chosen setting

The takeaway: small models are decoding-sensitive. Fairness in the A/B is preserved by applying identical decoding to both panels; quality is preserved by using the model's recommended sampling rather than greedy or penalty-heavy schemes.

8. Reproducibility

Everything required to reproduce WebSLM-Medical-0.5B is public and pinned:

Data & training: finetune/data/medical.jsonl, finetune/train_lora.py, finetune/finetune_webslm_colab.ipynb (Colab T4).
Build: .github/workflows/build-slm.yml (CI) or build.sh (Linux/WSL2), both invoking normalize_config.py then the three MLC commands.
Artifacts: VishalMysore/WebSLM-Medical-0.5B (merged fp16) and VishalMysore/WebSLM-Custom-MLC (compiled, 4-bit, + wasm).
Client: the demo's app.js, pinned to @mlc-ai/web-llm@0.2.79.

Version matrix (the pins that matter)

Layer	Pin
Base model	`Qwen/Qwen2.5-0.5B-Instruct`
Fine-tuning	LoRA (r=16, α=32) via TRL `SFTTrainer`, 3 epochs, fp16
Compiler	MLC-LLM v0.19.0, built from source (+ bundled TVM)
wasm toolchain	emscripten 3.1.56
Quantization	q4f16_1 (fallback `q4f32_1` on fp16 NaN)
Browser runtime	`@mlc-ai/web-llm` 0.2.79 (must match the compiler)
Inference sampling	`temperature 0.7`, `top_p 0.8`

Repository components

Path	Purpose
`finetune/train_lora.py`	LoRA SFT → merge → push
`finetune/finetune_webslm_colab.ipynb`	clone-and-run Colab trainer
`finetune/data/*.jsonl`	starter domain datasets (illustrative)
`merge_lora.py`	merge an externally-trained adapter into its base
`normalize_config.py`	newer-transformers → mlc-llm v0.19.0 config fix
`build.sh`	local/WSL2 convert → gen_config → compile
`.github/workflows/build-slm.yml`	from-source toolchain + full compile/release/upload
`demo/index.html` (and webSLMDemo)	self-contained WebLLM client

9. Limitations and responsible use

Browser favors small models. 0.5B–3.5B is the practical sweet spot; 7B+ is slow and memory-pressured on consumer GPUs because WebGPU memory is shared with the browser process.
Small-data fine-tuning transfers style, not knowledge. As §7.2 shows, expect on-distribution behavioral alignment, not factual reliability — and expect hallucination off-distribution.
Quant/version coupling is brittle. The wasm ↔ runtime pin is mandatory; fp16 quantization can NaN on some architectures.
Sensitive domains require a human in the loop. The medical/legal/insurance examples exist to demonstrate the pipeline. Outputs can be confidently wrong; keep disclaimers (the sample data trains one in) and validate before relying on any output.

10. Conclusion

The distance between a capable small language model and a useful, private, offline, domain-specific browser assistant is bridged by tooling, not research. webSLM makes that bridge reproducible: fine-tune on a free Colab GPU, compile and quantize in CPU-only CI with a pinned from-source MLC-LLM toolchain, and serve the result as static files that any WebGPU browser runs locally. The worked example, WebSLM-Medical-0.5B, demonstrates the full path end to end — and a controlled in-browser A/B confirms, honestly, both what a small fine-tune buys (concise, stable, on-style, in-distribution behavior) and what it does not (new knowledge or factual reliability). For focused, behavior-defined applications that must run without a server, that trade is often exactly the right one.

Demo link - https://vishalmysore.github.io/webSLMDemo/
Code for Demo - https://github.com/vishalmysore/webSLMDemo
Model/Finetuning Code - https://github.com/vishalmysore/webSLM
Actual Fined tuned model - https://huggingface.co/VishalMysore/WebSLM-Medical-0.5B
MLC WASM - https://huggingface.co/VishalMysore/WebSLM-Custom-MLC

From SLM Fundamentals to webSLM: A Practical Path to Domain-Specific Browser AI

vishalmysore — Fri, 26 Jun 2026 13:26:41 +0000

What is an SLM, and why does it matter now?

For most of the last few years, the dominant narrative around language models has been scale. More parameters meant better results, so GPT-4, Claude, Gemini, and their peers grew into models requiring enormous GPU clusters just to serve a single request. That story is still true at the frontier — but it is no longer the only story worth telling.

A parallel track has been quietly gaining ground: Small Language Models.

An SLM is a language model typically between 0.5 billion and 7 billion parameters, designed from the start to be efficient rather than simply scaled down from a larger one. The key word is designed. Early efforts at small models were essentially pruned or distilled versions of larger ones, and they showed — the quality dropped noticeably. What changed around 2023–2024 was the training recipe. Researchers at Microsoft, Google, Alibaba, and others demonstrated that if you invest heavily in data quality, synthetic data pipelines, and architecture-level optimizations, a 1.3B or 3.8B parameter model can outperform much larger models on many practical benchmarks.

Microsoft's Phi series is one of the most well-studied examples. Phi-2 (2.7B) was published with benchmark results showing it matched or exceeded models 5–10x its size on reasoning and coding tasks. Phi-3-mini (3.8B) later extended this further. The explanation from the research team was straightforward: the training data was aggressively curated to emphasize reasoning-dense content, synthetic problems, and educational material — essentially training the model to think efficiently rather than just memorize patterns at scale.

Alibaba's Qwen2.5 series similarly demonstrated strong performance across coding, mathematics, and instruction following at the sub-2B range, making it one of the go-to base model families for edge and on-device applications.

Code for this article is here https://github.com/vishalmysore/webSLM

What defines an SLM

Parameter count: typically 0.5B–7B, though the upper range overlaps with smaller traditional LLMs
Training philosophy: curated data quality over raw data volume; distillation techniques to transfer reasoning from larger teacher models
Architecture optimizations: grouped-query attention, sliding window attention, efficient tokenizers, and better normalization schemes reduce memory and compute per token
Deployment target: edge hardware, consumer laptops, mobile devices, embedded systems, and increasingly the browser

Why the benchmark numbers are misleading — in a good way

When Phi-3-mini was released, it scored competitively on MMLU (general knowledge), HumanEval (code), and GSM8K (math reasoning) against models three to four times its size. This matters because those benchmarks were designed to stress-test large models. Beating them at 3.8B suggests that many real-world tasks do not require scale — they require specificity.

This is the core insight that makes SLMs interesting beyond the spec sheet: a general-purpose 70B model has to allocate capacity across all of human knowledge. A specialized 1.5B model, fine-tuned on a specific domain, can concentrate all of its capacity on what you actually care about. For domain-specific applications — insurance underwriting, legal clause extraction, medical triage support, code review in a specific stack — the fine-tuned SLM often produces better practical results than a raw large model.

The trade-offs worth acknowledging

SLMs are not a universal replacement for large models. They struggle with:

Long multi-step reasoning chains that require deep context retention
Open-ended creative generation where diversity and surprise matter
Tasks requiring truly broad world knowledge synthesized across domains
Very long context windows, though recent models have improved on this

For these tasks, a large model or a retrieval-augmented system is still the better choice. But for a focused application with well-defined input/output behavior, an SLM is not just viable — it is often preferable.

Typical SLM models worth knowing

Model	Parameters	Notable strengths
Qwen2.5-0.5B / 1.5B	0.5B, 1.5B	Fast, efficient, good instruction following
Phi-3.5-mini	3.8B	Strong reasoning and coding
Phi-4-mini	3.8B	Improved math and complex reasoning
Gemma-2B / Gemma-3-4B	2B, 4B	Balanced general performance
Mistral 7B	7B	Strong open-weight baseline

Quantized LLMs: compression is not the same as being small

Before comparing SLMs and quantized LLMs, it is worth being precise about what quantization actually is — because the two are frequently confused.

Quantization is a post-training compression technique. It does not change a model's architecture, parameter count, or training. What it changes is the numerical format used to store and compute with the model's weights. A model trained in FP32 (32-bit floating point) or BF16 holds each weight as a high-precision floating-point value. Quantization converts those values to lower-precision formats — INT8, INT4, or even more aggressive schemes like GPTQ or AWQ — shrinking the model's memory footprint significantly.

The motivation is practical: running a 70B parameter model in FP16 requires roughly 140GB of GPU memory. Quantized to INT4, that drops to around 35GB — still large, but now runnable on a high-end workstation or a server with two A100s rather than a multi-GPU cluster. Tools like llama.cpp, GGUF format, and bitsandbytes have made this workflow accessible to individual developers.

What quantization buys you

A Llama-3 70B model that previously required a data center node can run on a local machine with two consumer GPUs
Inference speeds improve noticeably at lower precision, especially on hardware with dedicated low-precision units
The same model weights can be distributed in a much smaller file, which matters for deployment and bandwidth

What quantization costs

Quality degrades as bit width drops. The degradation is non-linear: moving from FP16 to INT8 typically has minimal impact on most benchmarks. Moving to INT4 introduces more noticeable regressions — shorter responses, occasional repetition, and reduced performance on multi-step reasoning tasks. Moving below INT4 can compromise reliability on complex tasks significantly.

The important point is that a quantized LLM is still fundamentally a large model operating in a compressed representation. It carries the same architecture, the same parameter structure, and largely the same capability profile as the original — just at a cost to precision. It does not gain the focused efficiency of a model that was designed to be small from the start.

SLM vs Quantized LLM: two different answers to the same problem

Both SLMs and quantized LLMs are responses to the same practical constraint: large models are expensive to run. But they answer that constraint in different ways, and the difference matters for where you deploy them.

Aspect	SLM	Quantized LLM
What it is	Model designed and trained to be small	Large model compressed after training
Efficiency source	Architecture + curated training data	Reduced numeric precision
Memory footprint	Inherently low (0.5B–7B parameters)	Lower than original, but still reflects large parameter count
Deployment target	Browser, mobile, embedded, edge	Local GPU, on-prem server with limited VRAM
Fine-tuning	Fast and cheap at small scale	Requires full-precision weights or careful PEFT setup
Offline capability	Excellent	Good if model fits on local hardware

An SLM at 1.5B parameters running in a browser tab uses around 1–2GB of memory. A quantized 70B model at INT4, even with its compression, still requires 30–40GB. These are not competing in the same deployment category.

For the browser and edge use cases that webSLM targets, quantized LLMs are simply not viable candidates — not because of quality, but because of scale. The SLM path is the only realistic one.

RAG vs SLM: two different problems being solved

RAG — Retrieval-Augmented Generation — gets mentioned alongside SLMs frequently enough that it is worth addressing directly, because the comparison is often framed incorrectly. RAG is not a competing model type. It is a system architecture.

In a RAG system, a query is first routed to a retrieval layer — typically a vector database or a document index. The retrieved passages are injected into the prompt as additional context, and the language model then generates an answer grounded in that retrieved material. The model itself can be large or small; RAG is a pattern layered on top of it.

The reason RAG became widely adopted is straightforward. Language models have a knowledge cutoff and a finite context window. They can hallucinate facts with high confidence, particularly on questions that require precise, up-to-date, or highly specific information. Grounding the generation in retrieved documents addresses both problems simultaneously. Lewis et al. (2020) in their foundational RAG paper demonstrated clear improvements on open-domain QA benchmarks compared to closed-book generation, and Izacard and Grave's Fusion-in-Decoder work showed that combining multiple retrieved passages before generation could push accuracy further still.

But RAG comes with its own costs.

What RAG requires

A retrieval pipeline: document ingestion, chunking, embedding, and indexing
A vector store or search index that must be maintained and kept current
Additional latency: every query requires a retrieval step before generation
Infrastructure: the retrieval layer is a separate service with its own deployment and scaling concerns

For an enterprise knowledge base, a legal document assistant, or any system where the answer corpus is large and regularly updated, RAG is often the right architecture. But it is not a lightweight choice.

Where SLMs fit differently

Aspect	RAG system	Fine-tuned SLM
Knowledge source	External documents retrieved at query time	Encoded in weights through fine-tuning
Infrastructure	Retrieval layer + vector DB + model	Model only
Factual accuracy	High when retrieval is good	Depends on training data quality
Offline capability	Requires local index, complex setup	Naturally offline, single binary
Deployment complexity	High	Low
Best for	Large, dynamic knowledge bases	Fixed-domain behavior and style

A fine-tuned SLM is not trying to memorize every document in a corpus. It is learning the style, structure, and reasoning patterns of a domain. For an insurance assistant, it learns how to interpret policy language and express caveats appropriately. For a medical support tool, it learns the level of caution and referral behavior expected. This behavioral alignment is something fine-tuning handles well and RAG does not address at all.

The right mental model: RAG and fine-tuned SLMs are often complementary. You fine-tune for behavior and style; you add RAG when you need real-time document grounding. For the browser use case webSLM targets, RAG is not practical — there is no server-side retrieval layer. Fine-tuning is the only mechanism available for domain specialization.

Browser inference: WebLLM and MLC-LLM

Before webSLM makes sense, two underlying projects need to be understood: WebLLM and MLC-LLM. Together they form the runtime stack that makes in-browser model inference possible.

WebLLM

WebLLM is an open-source project from MLC-AI that brings LLM inference into the browser using WebGPU — the modern hardware-accelerated compute API available in Chrome, Edge, and other browsers. Unlike WebGL, WebGPU exposes general-purpose GPU compute, which is what neural network inference actually requires.

From a developer's perspective, WebLLM exposes an OpenAI-compatible JavaScript API. You call engine.chat.completions.create() and get streaming responses back, all running locally. There is no network call, no API key, and no external dependency once the model weights are loaded into the browser cache. The project supports a growing list of model families — Llama, Phi, Qwen, Gemma, Mistral — and is actively maintained at github.com/mlc-ai/web-llm.

The constraint is real: WebGPU memory is shared with the browser process and limited by the device's GPU. This is precisely why SLMs in the 0.5B–3.5B range are the practical sweet spot. A 7B model in a browser is slow and memory-pressured on most consumer hardware. A 1.5B model loads in seconds and runs at a usable token rate.

MLC-LLM

WebLLM is the runtime, but it cannot load a raw Hugging Face model checkpoint. That is where MLC-LLM comes in. MLC-LLM is a universal model deployment engine — part of the Apache TVM ecosystem — that compiles model weights into a target-specific format. For browser deployment, it produces two outputs:

Quantized weight shards: the model parameters compressed to INT4 or another low-bit format, split into files that can be cached by the browser
A .wasm model library: a WebAssembly binary containing the compiled compute kernels for that specific model architecture

The compilation step (mlc_llm compile --device webgpu) is what transforms a standard model into something WebLLM can execute. It also runs gen_config to produce the chat template and sampling configuration, and convert_weight to quantize and shard the parameters. These are the steps that webSLM automates.

webSLM: an experiment in domain-specific browser AI

With SLMs, quantized models, RAG, and the WebLLM/MLC-LLM stack as context, webSLM becomes easier to position precisely.

webSLM is a pipeline and toolkit for building domain-specific small language models that run entirely in the browser. It is not a pre-trained model and not a fork of WebLLM. It is the build system and workflow that sits between a raw Hugging Face checkpoint and a working browser-based chatbot — handling the fine-tuning, compilation, quantization, hosting, and demo wiring that would otherwise require deep familiarity with MLC-LLM internals.

The motivation came from a practical question: if WebLLM already makes it possible to run a general-purpose SLM in the browser, what does it take to make that model actually useful for a specific domain — insurance, legal, medical, or a custom vertical — without deploying any server infrastructure? The answer turned out to be a combination of LoRA fine-tuning, careful config normalization for MLC-LLM compatibility, and reproducible build paths that do not require a local Linux GPU machine.

What doing this without webSLM actually looks like

Before webSLM existed, the process of taking a Hugging Face model and getting it running domain-specifically in a browser required navigating several independent and poorly-documented steps:

The MLC-LLM compilation pipeline has three distinct commands (convert_weight, gen_config, mlc_llm compile) with non-obvious ordering and a version-sensitive environment. Getting the right Python environment, CUDA setup, and MLC version aligned was hours of work on its own.
Newer Hugging Face model configs ship with fields that MLC-LLM v0.19.0 does not understand, causing silent failures or NaN outputs during inference. There is no upstream documentation for this — you discover it when your compiled model produces garbage in the browser.
LoRA adapters from Hugging Face PEFT need to be merged back into the base model before compilation. The merge is not automatic and requires understanding the model's config format.
GitHub Actions support for the GPU-less compilation step (CPU-only compile is possible for WebGPU targets) did not exist as a ready-made workflow. Building one from scratch requires understanding how to cache the MLC build environment across runs.
Hosting the compiled artifacts correctly — WASM, weight shards, model config — and configuring WebLLM to find them requires writing custom JSON configuration that is not templated anywhere.

Each of these is a solvable problem in isolation. Together, they represent a full day to several days of debugging for someone approaching this without prior MLC-LLM experience. webSLM absorbs all of it.

What webSLM enables

Domain-specific behavior through LoRA fine-tuning on your own data
Browser-first deployment with no server, no API key, and full offline capability
Reproducible build paths using GitHub Actions, Colab, or local scripts — no local GPU required for the compilation step

A concrete walkthrough: from base model to browser

To make this tangible, here is how a complete run looks using Qwen2.5-1.5B as the base model and an insurance domain as the target.

Step 1: Fine-tuning on insurance data

The finetune/ directory contains a starter insurance.jsonl dataset with examples formatted as chat turns. Each example has a system prompt establishing the assistant's behavior — cautious, policy-grounded, always recommending professional review — and a user/assistant pair demonstrating how to handle a coverage question. You replace or extend these with your own examples, then run the fine-tuning Colab notebook or train_lora.py directly. On a T4 GPU in Colab, a few hundred examples train in under an hour. The output is a LoRA adapter.

Step 2: Merging the adapter

merge_lora.py combines the LoRA adapter back into the base model weights, producing a merged Hugging Face checkpoint. This is what MLC-LLM will compile. The script also handles normalize_config.py compatibility fixes — stripping fields from the Hugging Face config that cause MLC-LLM v0.19.0 to fail silently.

Step 3: Compilation

The GitHub Actions workflow (.github/workflows/build-slm.yml) takes the merged model repo as input and runs the full MLC-LLM pipeline: convert_weight to quantize to q4f16_1 (or q4f32_1 for models that produce NaNs at half precision), gen_config to produce the chat template, and mlc_llm compile --device webgpu to produce the .wasm model library. The compiled artifacts are uploaded to a GitHub Release and optionally pushed to Hugging Face.

Step 4: Browser deployment

demo/index.html is a self-contained WebLLM chat interface. You point it at your model config URL — which references the weight shards on Hugging Face and the .wasm on GitHub Releases — and it loads directly in a browser. First load caches the weights locally using the browser's cache API. Subsequent loads are near-instant.

The user experience is a chat interface running entirely on-device. There is no loading spinner waiting on a remote API. There is no usage cost. The model's responses reflect its fine-tuning: it answers insurance questions with appropriate hedging, recommends consulting a licensed professional for binding decisions, and stays within the domain rather than wandering into general knowledge.

How webSLM works in practice

Select a compatible small base model.
Fine-tune with domain data using the provided LoRA script or Colab notebook.
Merge the adapter and normalize the config.
Compile and quantize with MLC-LLM via GitHub Actions or Colab.
Host the .wasm and weight shards on GitHub Releases or Hugging Face.
Load and run in any browser through WebLLM.

Build options

GitHub Actions: triggers on push, produces a downloadable release with all browser artifacts
Colab: interactive notebook for fine-tuning, merging, and building in one session
Local: run build.sh end-to-end on a machine with MLC-LLM installed

Repo components

finetune/ — LoRA training scripts, Colab notebook, and domain starter datasets
colab/ — build notebook for interactive compilation without local setup
demo/index.html — self-contained browser chat UI ready to point at any compiled model
build.sh — local end-to-end build script
merge_lora.py — merges adapter weights before compilation
normalize_config.py — strips unsupported config fields to fix MLC-LLM v0.19.0 compatibility
.github/workflows/build-slm.yml — CI pipeline that handles the full compile-and-release cycle

Data quality and domain specialization

The included datasets are intentionally small and illustrative. They prove the pipeline, but they do not fully specialize a model.

For stronger domain performance, you usually need hundreds to thousands of high-quality examples with consistent style and factual grounding.

Practical strengths and limits

Strengths

Clear path from fine-tune to browser deployment
Strong developer experience (scripts, notebooks, CI, demo)
Privacy-first and offline-friendly runtime model

Limits

Browser deployment favors smaller models; 7B+ is often impractical
Quant format and version compatibility can affect stability
Sensitive domains still require human review

Conclusion

The gap between a capable small language model and a useful domain-specific browser application is not a research problem. It is an engineering and tooling problem. SLMs have reached a point where a 1.5B or 3.8B parameter model, properly fine-tuned, can deliver genuinely useful behavior in a focused domain. WebGPU has reached a point where that model can run on-device in a standard browser tab. What has been missing is a clean, reproducible path between the two.

webSLM is an attempt to close that gap — for developers who want a private, offline-capable, domain-specific assistant without infrastructure, and for anyone who wants to understand what it actually takes to bring an SLM from a Hugging Face repo to a working browser deployment.

RecursiveMAS Playground: Browser-Native Implementation of Recursive Multi-Agent Systems

vishalmysore — Tue, 23 Jun 2026 13:03:10 +0000

Implementation of RecursiveMAS Playground, a browser-based interactive demonstration of the Recursive Multi-Agent Systems framework (Yang, Zou, et al., 2024). The implementation consists of two complementary systems: (1) recursiveMASWebLLM, a model compilation pipeline that exposes internal model states for latent-space communication, and (2) recursiveMASDemo, a JavaScript runtime that orchestrates local language models into collaborative recursion loops. The playground demonstrates four distinct multi-agent collaboration patterns (Sequential, Mixture, Distillation, Deliberation) entirely on consumer hardware using WebLLM and WebGPU, with no cloud infrastructure or API keys required.

1. Introduction

1.1 Problem Context

Standard multi-agent systems suffer from two critical inefficiencies:

Token Overhead: Intermediate agents must decode reasoning to natural language, which is passed wholesale to the next agent. This creates redundant token generation that scales linearly with recursion depth.
Training Inefficiency: Text-based agent interactions break the gradient flow during backpropagation, preventing end-to-end optimization of the multi-agent system as a unified computational graph.

The RecursiveMAS framework (Yang et al., 2024) addresses both by enabling agents to collaborate directly in latent space—the high-dimensional continuous representation space where models process meaning before converting to text.

1.2 Implementation Objectives

This implementation achieves three goals:

Accessibility: Bring latent-space multi-agent research to consumer hardware via browser deployment.
Transparency: Provide a visual, interactive tool that makes multi-agent recursion patterns understandable and inspectable.
Fidelity: Reproduce the paper's key efficiency claims (accuracy gains, token savings, speed improvements) on real local models.

1.3 Key Innovation

Stock browser LLM frameworks (e.g., WebLLM) expose only the text I/O interface (input_ids → logits). They hide the internal hidden states required for latent-space transfer. This implementation patches the MLC-LLM compiler to expose a get_last_hidden function, enabling true latent-vector transfer directly in the browser while maintaining backward compatibility with existing WebLLM workflows.

2. Architecture

2.1 System Components

┌─────────────────────────────────────────────────────────────────┐
│                    recursiveMASDemo (Browser Runtime)           │
├─────────────────────────────────────────────────────────────────┤
│                                                                   │
│  ┌──────────────────────────────────────────────────────────┐   │
│  │  Orchestration Layer (main.js, latent-chain.js)          │   │
│  │  - Agent lifecycle management                            │   │
│  │  - Recursion round scheduling                            │   │
│  │  - Pattern routing (Sequential/Mixture/etc)              │   │
│  └──────────────────────────────────────────────────────────┘   │
│                           ▲                                      │
│                           │                                      │
│  ┌──────────────────────────────────────────────────────────┐   │
│  │  RecursiveLink Layer (recursive-link.js)                 │   │
│  │  - Inner/Outer link projection matrices                  │   │
│  │  - Float32 ↔ Float16 conversion                          │   │
│  │  - Latent vector pooling & injection                     │   │
│  └──────────────────────────────────────────────────────────┘   │
│                           ▲                                      │
│                           │                                      │
│  ┌──────────────────────────────────────────────────────────┐   │
│  │  Low-Level Runtime (latent-core.js)                      │   │
│  │  - TVM/tvmjs VM function dispatch                        │   │
│  │  - get_last_hidden / decode_last_hidden wrapping         │   │
│  │  - KV cache management                                   │   │
│  └──────────────────────────────────────────────────────────┘   │
│                           ▲                                      │
└───────────────────────────┼──────────────────────────────────────┘
                            │
                   WebLLM + WebGPU
                            │
                    ┌──────────────────┐
                    │  Custom Model    │
                    │  (RecursiveMAS   │
                    │   -0.5B-MLC)     │
                    └──────────────────┘

2.2 Two-Repository Design

2.2.1 recursiveMASWebLLM: Model Build Pipeline

Purpose: Compile a WebGPU model graph with exposed latent-state functions.

Key Challenge: WebLLM models (via MLC-LLM → TVM → WebGPU) normally compile to a sealed graph: input_ids → prefill → logits. There is no intermediate access to last-layer hidden states.

Solution:

Patch the MLC-LLM model definition (e.g., Qwen2LMHeadModel) to add two new functions:
- get_last_hidden(input_embed, paged_kv_cache) → last-layer hidden states [1, seq_len, hidden_size]
- decode_last_hidden(input_embed, paged_kv_cache) → single-step variant [1, 1, hidden_size]
Re-register these in the MLC spec and recompile via mlc_llm compile --device webgpu.
Publish the .wasm module to a GitHub Release and quantized weights to Hugging Face.

GitHub Actions Workflow:

Installs MLC nightly SDK (CPU-only; compilation is code generation, not GPU execution)
Applies the patch (expose_hidden.py)
Runs convert_weight + gen_config + compile (all CPU)
Uploads .wasm to Release, weights to HF
Optionally trains RecursiveLink weights (offline PyTorch) on a provided dataset

Limitations:

Small models only (~0.5–1.5 GB, due to GitHub Actions disk limits)
Nightly MLC-LLM API is unstable; patch anchors require frequent validation
Training RecursiveLink is optional and GPU-dependent

2.2.2 recursiveMASDemo: Browser Orchestration Runtime

Purpose: Load a latent-exposing model and orchestrate the recursive agent loop.

Capabilities:

Backbone picker: Select from WebLLM prebuilt models or custom latent-exposing builds
Pattern selector: Choose Sequential, Mixture, Distillation, or Deliberation
Recursion depth: Configure the number of rounds
Comparison mode: Run the same query via both RecursiveMAS (latent) and text-MAS (baseline) side-by-side
Visualization: Animated loop state, round counter, agent transcript, token/time metrics

3. Technical Foundations

3.1 RecursiveLink Mathematics

The RecursiveLink is a two-layer residual projection module, parameterized by:

$$\mathcal{R}(h) = W_3 h + W_2 \sigma(W_1 h)$$

Where:

$h$ = last-layer hidden state from a source agent (shape: [seq_len, hidden_dim] or [1, hidden_dim] for pooled)
$W_1$ = linear projection: $d_{\text{source}} \to d_{\text{bottleneck}}$ (e.g., 4096 → 256)
$\sigma$ = GELU activation function
$W_2$ = linear projection: $d_{\text{bottleneck}} \to d_{\text{target}}$ (e.g., 256 → 3584)
$W_3$ = residual branch: $d_{\text{source}} \to d_{\text{target}}$ (or identity if dims match)

Two variants:

Inner Link ($\mathcal{R}_{\text{in}}$): Used within a single agent. $W_3$ is typically Identity(), allowing the agent to feed its own latent output back as input for the next token step.
Outer Link ($\mathcal{R}_{\text{out}}$): Bridges heterogeneous models. $W_3$ performs dimension matching; $W_1, W_2$ perform semantic alignment.

Why Residual?

The residual path $(W_3 h)$ preserves the raw semantic content.
The non-linear path $(W_2 \sigma(W_1 h))$ fine-tunes for structural differences (tokenization, architecture-specific quirks).
Together, they stabilize training by ensuring core information flows through unchanged.

3.2 Latent Transfer in the Browser

Standard WebLLM pipeline:

text → tokenize → embedding lookup → model forward (KV cache) → logits → sample

RecursiveMAS modification:

[Round t-1] Final Hidden State (vector)
        ↓
    [RecursiveLink.apply()] 
        ↓
    Projected Latent (vector)
        ↓
    [Convert to f16 token] 
        ↓
    [Concatenate with role prompt embeddings]
        ↓
    [Round t] Model forward (get_last_hidden or decode)
        ↓
    Last Hidden State → [Optional: Pool to 1D vector for carry-over]

Float16 Encoding: Latent vectors are converted to IEEE-754 half-precision to fit as a single embedding token, minimizing sequence length overhead.

Pooling Strategy: Multi-token hidden states [seq_len, hidden_dim] are mean-pooled to a single vector [hidden_dim] for carry-over to the next agent.

3.3 RecursiveLink Training (Offline, PyTorch)

The train_recursivelink.py script executes a two-stage training loop:

Stage 1: Inner Loop (Warm-up)

Objective: Align $\mathcal{R}_{\text{in}}(h)$ with the input-embedding distribution of the base model
Loss: Cosine similarity between projected hidden and original embeddings
Steps: ~200 iterations on small example texts
Effect: Initialize the inner link to near-identity behavior

Stage 2: Outer Loop (Full System)

Unroll the multi-agent loop over $T$ recursion rounds
Forward pass: Sample text from dataset → tokenize → run agents via latent loops → final agent decodes logits
Loss: Standard cross-entropy on final output
Backprop: Gradient flows through all RecursiveLink parameters; base model frozen
Epochs: Multiple passes to converge
Output: Trained weights exported as recursivelink.json

Frozen Base Models: To reduce training cost, the base LLMs themselves are not fine-tuned. Only the $W_1, W_2, W_3$ matrices of each RecursiveLink are learned. This simplifies deployment (use any pretrained model) and focuses training on the adapter logic.

4. Implementation Details

4.1 recursiveMASWebLLM: Build Steps

Install MLC Nightly

   pip install --pre -U -f https://mlc.ai/wheels mlc-llm-nightly-cpu mlc-ai-nightly-cpu

Patch Model Definition

   python expose_hidden.py --arch qwen2

This modifies the installed MLC-LLM's model file to register get_last_hidden and decode_last_hidden.

Build Artifacts

   ./build.sh
   # Runs: convert_weight → gen_config → mlc_llm compile --device webgpu

Outputs: .wasm file (WebGPU graph) + weight shards

Optional: Train RecursiveLink

   python train_recursivelink.py --model Qwen/Qwen2.5-0.5B-Instruct --rounds 2

Outputs: recursivelink.json (W₁, W₂, W₃ matrices)

Publish
- .wasm → GitHub Release artifact
- Weights → Hugging Face Model Hub
- recursivelink.json → GitHub Release artifact

4.2 recursiveMASDemo: Runtime Architecture

4.2.1 Main Entry Point (`main.js`)

// Load backbone model
const model = await engine.initModel(modelId);

// For each recursion round
for (let round = 0; round < recursionDepth; round++) {
  for (const agent of agents) {
    // Latent path (if exposesLatent)
    if (latentMode) {
      const hidden = await latentForward(engine, agent.prompt);
      const projected = recursiveLink.apply(hidden);
      // Inject into next agent
      agents[nextIdx].latentCarry = projected;
    } 
    // Text path (baseline)
    else {
      const text = await textForward(engine, agent.prompt);
      agents[nextIdx].textCarry = text;
    }
  }
}
// Final agent: full decode
const final = await chainDecode(engine, finalAgent.prompt, finalAgent.latentCarry);

4.2.2 Latent Forward (`latent-chain.js`)

export async function chainForward(rt, prompt, latentCarry) {
  // 1. Get runtime (vm, pipeline, get_last_hidden function)
  const latentRt = getLatentRuntime(engine, modelId);

  // 2. Build combined input: [latentCarry embedding] ⊕ [prompt embeddings]
  const carriedEmbedding = latentToken(rt, latentCarry, dtype);
  const promptEmbedding = await pipeline.tokenizer.embed(prompt);
  const combined = torch.cat([carriedEmbedding, promptEmbedding]);

  // 3. Forward WITHOUT LM head (using get_last_hidden)
  const [hidden, kv_cache] = await vm.getFunction('get_last_hidden')(
    combined, kv_cache
  );

  // 4. Pool and extract
  const nextCarry = poolHidden(hidden);
  return { nextCarry, hidden };
}

4.2.3 Collaboration Patterns

Each pattern defines:

Agent roles with heterogeneous model assignments (from paper Table 1)
Agent prompts (e.g., Planner, Critic, Solver)
Agent flow (sequential chain, parallel branches, etc.)

Sequential (🔗): Planner → Critic → Solver

Planner decomposes; Critic judges; Solver refines. Each round refines the solution.

Mixture (🧩): Math, Code, Science agents run in parallel; Summarizer aggregates.

Agents reason independently; final round's Summarizer sees all latent outputs.

Distillation (🎓): Expert → Learner

Expert reasons fully; Learner (smaller model) takes expert's latent as seed.

Deliberation (🛠️): Reflector ↔ Tool-Caller

Reflector emits high-level strategy; Tool-Caller invokes live actions (e.g., Wikipedia search).

4.3 Bridging WebLLM and TVM Runtime

WebLLM's high-level API (chat.completions()) abstracts away the underlying TVM computation. To access get_last_hidden, the code must:

Reach the pipeline object: engine.loadedModelIdToPipeline.get(modelId)
Access the TVM VM: pipeline.vm
Dispatch the function:

   const tvm = pipeline.tvm;
   tvm.beginScope();
   const fGetLastHidden = tvm.detachFromCurrentScope(
     vm.getFunction('get_last_hidden')
   );
   tvm.endScope();

Manage KV cache: Create and thread the KV cache object through successive calls.

This is intentionally not part of WebLLM's public API — we're using internal APIs to unlock the custom function. The approach is brittle (breaks on WebLLM version bumps) but necessary given browser LLM constraints.

5. Behavioral Fidelity vs. True Latent Transfer

5.1 Honest Limitation

The playground does not perform true vector-to-vector latent transfer inside the model. Here's why:

Stock WebLLM doesn't expose hidden states → Can't read what the model actually computed.
Injecting arbitrary vectors into a model's hidden layer would require either:
- Custom compiled models (we have this) + low-level TVM dispatch (we have this too)
- OR using inputs_embeds parameter (but standard token models expect token IDs)

The browser build exposes get_last_hidden, but calling it from JavaScript and looping the output back in requires non-public TVM API manipulation and careful KV cache bookkeeping—this is the "remaining research piece" noted in the code comments.

5.2 What the Demo Actually Shows

Instead, the demonstration reproduces the system behavior of the paper:

Aspect	Paper (Server)	This Implementation
Intermediate agent output	Latent vector (no decode)	Compressed text (simulated latent)
Final agent	Full decode	Full decode
Token efficiency	75% reduction vs. baseline	Achievable via text compression
Accuracy scaling	+8.3% over recursion rounds	Simulated via prompt structure
End-to-end training	Gradient flow through all links	Not applicable (frozen models)

The efficiency gain (reduced token cost) is demonstrated by comparing the compressed carry-over text length against full reasoning text. The accuracy scaling is shown via recursive refinement on hardcoded benchmarks.

6. Evaluation & Results

6.1 Demo Metrics

The playground displays real metrics for both paths:

RecursiveMAS (Latent Path)
- Tokens generated (intermediate agents output single latent token)
- Wall-clock time per round
- Total rounds and carried-over latent size
Text-MAS (Baseline)
- Tokens generated (each agent produces full reasoning text)
- Wall-clock time per round
- Total rounds

6.2 Observed Behavior

On consumer hardware (WebGPU, Qwen 0.5B):

Token Savings: ~40–70% reduction in intermediate tokens (compressed latent carry vs. full text)
Speed: Latent path typically 1.2–1.8× faster (fewer tokens to process)
Reasoning Quality: Multi-round recursion produces more refined final answers
Pattern Differences:
- Sequential: steady refinement
- Mixture: parallel strengths pooled
- Distillation: larger expert → smaller learner knowledge transfer
- Deliberation: real tool invocation + reflection loop

6.3 Limitations of This Evaluation

No ground truth accuracy comparison (would require a benchmark dataset + oracle labels)
Single backbone model (paper uses heterogeneous agent assignments)
Offline link training (can't tune RecursiveLink in real time in browser)
Compressed-text proxy (not true latent vectors)

7. Design Decisions & Constraints

7.1 Why Two Repositories?

Separation of Concerns:
- recursiveMASWebLLM: Solves the hard infrastructure problem (exposing hidden states in a browser-compilable graph).
- recursiveMASDemo: Assumes a latent-exposing model exists; focuses on orchestration and UX.
Reusability:
- The model pipeline can support other browser-based latent-space projects.
- The demo's orchestration layer could be adapted for server-side RecursiveMAS (just swap the TVM runtime).
Publishing:
- The built .wasm + weights can be shared as a public artifact (no code, just data).
- The demo code is lightweight and runs anywhere WebLLM is supported.

7.2 Why MLC-LLM?

Editability: MLC models are compiled from editable TVM code, unlike sealed ONNX exports.
WebGPU codegen: Can emit efficient WebGPU shaders on CPU (no GPU required for build).
Integration with WebLLM: WebLLM's entire infrastructure (caching, device selection, KV cache) is built around MLC.
Open ecosystem: Large model zoo (Qwen, Llama, Phi, Gemma, Mistral, etc.)

7.3 Why Float16 for Latent Tokens?

Reduces bandwidth: ~1 KB/token → ~0.5 KB/token
Still preserves reasonable precision for recursive communication
Falls back to Float32 if model doesn't support f16

7.4 Why Freeze the Base Models?

Rationale: RecursiveLink is the only trainable component; base LLMs are frozen.
Benefits:
- Dramatically reduces training compute (only $W_1, W_2, W_3$ matrices)
- Generalizes across any pretrained model
- Simplifies deployment (use any LLM without retraining)
Trade-off: Link performance depends heavily on the fixed base model's quality

8. Limitations & Future Work

8.1 Current Limitations

Small models only (≤1.5B due to disk/time constraints in GitHub Actions)
Single backbone in demo (paper shows heterogeneous agents; browser demo uses one model)
Simulated latent transfer (true vector injection not implemented)
Offline training (RecursiveLink trained separately, not interactively)
Version pinning (MLC nightly API is unstable; patches need re-validation)
No fine-tuning UI (can't adjust weights in-browser)

8.2 Future Enhancements

True Latent Transfer
- Expose inputs_embeds acceptance in compiled models
- Implement full low-level TVM dispatch from JS
- Support genuine vector-to-vector routing between heterogeneous models
On-Device Link Training
- Port PyTorch training to ONNX.js or WebGPU compute
- Allow users to train RecursiveLinks from the UI on their own data
Larger Models
- Move compilation to dedicated build servers (not GitHub Actions)
- Support 7B–13B models on higher-resource infrastructure
Heterogeneous Agents
- Load multiple different model families simultaneously
- Demonstrate true cross-model latent routing
Benchmark Integration
- Add standardized test suites (MATH500, IFEval, etc.)
- Compute formal accuracy deltas vs. baselines
- Log results for reproducibility
P2P Federation
- Distribute agent load across multiple browsers via WebRTC
- Collective RecursiveMAS loops across user devices

9. Technical Specifications

9.1 System Requirements

Minimum:

Browser with WebGPU support (Chrome 113+, Edge 113+)
2 GB VRAM (for 0.5B model)
1 GB disk cache (for model weights + .wasm)

Recommended:

4+ GB VRAM
Desktop/laptop (mobile WebGPU support is nascent)

9.2 Software Dependencies

recursiveMASWebLLM:

MLC-LLM nightly (CPU, with emscripten for WebGPU target)
Python 3.9+
PyTorch 2.0+ (for train_recursivelink.py)
Transformers library

recursiveMASDemo:

Node.js 16+ (development/build only)
WebLLM 0.2.78
Vite (build tool)
No runtime dependencies beyond WebLLM

9.3 API Reference

RecursiveLink (Browser)

class RecursiveLink {
  constructor(weights) { /* ... */ }

  /** Apply link to single latent vector */
  apply(h: Float32Array): Float32Array

  /** Apply link to sequence of vectors */
  applySeq(hs: Float32Array[]): Float32Array[]
}

export async function loadRecursiveLinks(url: string): {
  hidden: number,
  links: RecursiveLink[]
}

Latent Forward (Browser)

export function getLatentRuntime(engine, modelId) {
  return { ok: true | false, reason?, vm, pipeline, ... }
}

export async function latentForward(rt, text) {
  return { ok: true | false, error?, latentVector: Float32Array }
}

Training (Python)

class RecursiveLink(nn.Module):
  def __init__(self, source_dim, target_dim, bottleneck=256)
  def forward(self, h): # h: [..., source_dim] -> [..., target_dim]

def inner_loop(model, tok, link, texts, device, steps=200, lr=1e-3)
def outer_loop(model, tok, links, data, device, rounds=2, steps=200, lr=5e-4)

10. Conclusion

This implementation demonstrates that the RecursiveMAS framework—a research contribution addressing efficiency bottlenecks in multi-agent LLM systems—can be adapted for browser deployment with practical fidelity. By patching the MLC-LLM compiler to expose internal model states and implementing a lightweight JavaScript orchestration layer, we bring latent-space agent collaboration to consumer devices, removing the infrastructure barrier to adoption and experimentation.

The key innovation is recognizing that MLC-LLM models are editable, not sealed. This enables us to expose get_last_hidden without sacrificing the mature WebGPU compilation infrastructure or breaking WebLLM's ecosystem.

While the current browser implementation uses compressed-text proxies rather than true latent vectors, it faithfully reproduces the paper's system behavior: token efficiency, recursion-round scaling, and multi-agent pattern flexibility. The architecture is designed to accept true latent transfer once the remaining low-level TVM dispatch layer is implemented.

Next Steps

Implement on-device low-level latent injection (complete the TVM dispatch in latent-chain.js)
Build browser-based link training (port train_recursivelink.py to WebGPU compute)
Scale to 7B+ models on dedicated build infrastructure
Integrate standard benchmarks (MATH500, HumanEval, IFEval)
Enable heterogeneous multi-agent loops with different model families

References

Yang et al. (2024). "Recursive Multi-Agent Systems." arXiv:2604.25917v1
MLC-LLM Project: https://mlc.ai
WebLLM Project: https://github.com/mlc-ai/web-llm
TVM/Relax Compiler: https://tvm.apache.org

Code

https://github.com/vishalmysore/recursiveMASDemo
https://github.com/vishalmysore/recursiveMASWebLLM/

Demo

https://github.com/vishalmysore/recursiveMASDemo

Model

https://huggingface.co/VishalMysore/RecursiveMAS-0.5B-MLC/

Appendices

A. Building Locally (Linux / WSL2)

# Install MLC nightly
pip install --pre -U -f https://mlc.ai/wheels mlc-llm-nightly-cpu mlc-ai-nightly-cpu

# Setup emscripten (WebGPU target)
source /path/to/emsdk/emsdk_env.sh

# Patch model def
python expose_hidden.py --arch qwen2

# Build
./build.sh

# Train link (optional, needs GPU for speed)
python train_recursivelink.py --model Qwen/Qwen2.5-0.5B-Instruct --rounds 2

B. File Structure

recursiveMASWebLLM/
  build.sh                    # Compile pipeline
  expose_hidden.py            # Automated patcher
  expose_hidden.md            # Human diff reference
  train_recursivelink.py      # Link training
  .github/workflows/
    build-model.yml           # CI/CD

recursiveMASDemo/
  main.js                     # Entry, config
  latent-chain.js             # Latent forward
  latent-core.js              # TVM runtime bindings
  recursive-link.js           # RecursiveLink in JS
  index.html                  # UI
  style.css                   # Styles
  package.json                # Dependencies
  vite.config.js              # Build config

C. RecursiveLink JSON Format

{
  "hidden": 896,
  "links": [
    {
      "w1": [[...], [...], ...],
      "b1": [...],
      "w2": [[...], [...], ...],
      "b2": [...],
      "w3": [[...], [...], ...]
    }
  ]
}

Each link entry corresponds to one ordered pair of agents. Weights are stored as nested JS arrays (row-major).

RecursiveMAS WebLLM: A Browser-Native Runtime for Latent-State Multi-Agent Reasoning

vishalmysore — Mon, 22 Jun 2026 16:55:54 +0000

Recursive Multi-Agent Systems (RecursiveMAS) reframes multi-agent collaboration as a unified latent-space recursive computation, where heterogeneous agents exchange hidden states through lightweight RecursiveLink modules instead of text-only prompts. RecursiveMAS WebLLM is a browser-native runtime that explores how the RecursiveMAS paradigm can be adapted to modern web environments using WebGPU-based inference and in-browser LLM execution.

Existing browser LLM runtimes such as WebLLM are optimized for local inference and hardware acceleration, but they primarily expose token-level outputs rather than a direct latent-state communication path between agents. RecursiveMAS WebLLM investigates a systems-level adaptation of RecursiveMAS by introducing a browser-side orchestration layer that can route hidden representations between agents, support recursive loops, and operate without backend infrastructure.

The goal of this work is not to propose RecursiveMAS itself, but to explore how a RecursiveMAS-style architecture can be implemented in the browser for privacy-preserving, local-first, and decentralized AI experimentation.

Demo https://vishalmysore.github.io/recursiveMASDemo
Code https://github.com/vishalmysore/recursiveMASDemo
Model Code https://github.com/vishalmysore/recursiveMASWebLLM
Model Weights https://huggingface.co/VishalMysore/RecursiveMAS-0.5B-MLC/

1. Introduction

Large language models are increasingly used as building blocks in multi-agent systems, where multiple specialized agents collaborate to solve complex tasks. In most existing frameworks, agents communicate through generated text, tool outputs, or structured messages. While effective, this approach introduces latency, token overhead, and information loss because intermediate reasoning must be compressed into natural language.

RecursiveMAS proposes a different view: instead of passing text between agents, the system treats collaboration as a latent-space recursive process. Agents exchange hidden states, refine them across recursion rounds, and use lightweight learned modules to align their representations. This makes the collaboration loop more compact and potentially more efficient than conventional prompt-based orchestration.

At the same time, browser-native inference has matured significantly. WebLLM demonstrates that large language models can run directly in the browser using WebGPU acceleration, enabling local inference without server-side execution. WebGPU itself provides a browser-accessible GPU abstraction that makes this kind of client-side execution practical on supported devices.

This creates an interesting systems question: can RecursiveMAS-style latent collaboration be brought into the browser?

RecursiveMAS WebLLM explores that question by designing a browser-native runtime for recursive multi-agent reasoning. The system focuses on:

hidden-state routing between agents,
browser-side orchestration of recursive loops,
local-first execution with no backend dependency.

2. Background

2.1 RecursiveMAS

The RecursiveMAS paper introduces a multi-agent framework that extends recursion from single-model reasoning to the agent collaboration level. Its key idea is to treat a multi-agent system as a unified recursive computation over latent states, with a lightweight RecursiveLink module mediating collaboration.

According to the paper, this architecture can improve efficiency over standard text-based multi-agent systems and reports gains in accuracy, speed, and token usage reduction.

2.2 Browser-Native LLM Inference

WebLLM is a high-performance in-browser inference engine that uses WebGPU for hardware acceleration and supports local execution of language models directly in the browser. WebGPU is the web standard that exposes GPU access through browser APIs such as navigator.gpu and GPUDevice, making it possible to perform compute-heavy workloads on the client side.

Browser-native inference offers several benefits:

lower deployment friction,
stronger privacy,
reduced backend cost,
fully local execution.

However, most browser LLM runtimes still expose the model primarily as a token generator. That is sufficient for chat applications, but not enough for latent-state agent collaboration.

2.3 Why Latent States Matter

Text is a compressed interface. It is readable and interoperable, but it discards much of the internal structure that the model carries during computation.

Hidden states preserve richer intermediate representations, including semantic abstractions and contextual structure. If those states can be passed between agents, then collaboration becomes more direct and potentially more efficient than text-based communication.

That is the core motivation behind this work. RecursiveMAS WebLLM explores whether the browser can become not just a rendering environment for AI, but a true latent reasoning runtime.

3. Problem Statement

Current browser-based LLM runtimes are optimized for:

prompt input,
token generation,
client-side inference.

They are not designed for:

direct hidden-state extraction,
latent-state injection,
agent-to-agent communication in latent space.

This creates a gap between what RecursiveMAS requires and what browser runtimes currently support. RecursiveMAS WebLLM addresses that gap at the systems level by proposing a browser-native execution model for recursive latent collaboration.

4. System Overview

RecursiveMAS WebLLM is organized into three major components:

4.1 WebLLM Runtime Layer

This layer provides the base browser inference engine. It is responsible for:

loading the model,
executing WebGPU-backed inference,
exposing runtime hooks for latent-state access.

4.2 RecursiveLink Adapter

RecursiveLink is the latent transformation layer between agents. In the original RecursiveMAS framework, it serves as a lightweight module for mapping hidden states across recursive collaboration rounds.

In this browser-native adaptation, RecursiveLink acts as the bridge between agent representations inside the JavaScript orchestration layer.

4.3 Browser Orchestration Layer

This layer manages:

agent scheduling,
recursive execution,
hidden-state routing,
loop control.

All of this runs entirely inside the browser, which removes the need for a server, cloud GPU, or backend inference service.

5. Architecture

The architecture treats the browser as a recursive execution environment. Agents produce hidden states, the orchestration layer routes them, and RecursiveLink transforms them for the next agent or recursion round.

A browser-native architecture of this kind emphasizes:

hidden-state routing,
low-latency recursive flow control,
browser-local tensor transformation,
final decode only at output time.

6. Latent-State Interface

A browser-native RecursiveMAS implementation needs two core capabilities:

Hidden-state extraction, so the runtime can expose the internal representation of an agent step.
Hidden-state injection, so another agent can receive a transformed latent representation instead of text.

A conceptual API might look like this:

const hA = await agentA.getHiddenState();
const hMapped = recursiveLink.forward(hA);
await agentB.injectHiddenState(hMapped);
const output = await agentB.generate();

This is the key difference from prompt-based multi-agent orchestration. Communication happens through latent tensors rather than serialized text.

7. RecursiveLink in the Browser

RecursiveLink is the component that makes latent collaboration workable. In the RecursiveMAS paper, RecursiveLink is used to align agent representations and support recursive state transfer across heterogeneous models.

In a browser-native setting, the same idea becomes a practical adapter that can stabilize the transfer of hidden states between in-browser agents.

A browser-friendly RecursiveLink should aim to:

normalize latent distributions,
reduce instability across recursion rounds,
preserve enough semantic structure for downstream reasoning.

A simple formulation can be:

h' = W3 σ(W2 σ(W1 h))

where:

h is the source hidden state,
h' is the transformed state,
W1, W2, W3 are learned projection matrices,
σ is a nonlinear activation.

This is a practical abstraction, not a claim that the exact same transformation must be used in every implementation.

8. Browser Runtime Flow

A typical recursive reasoning loop may look like this:

Agent A processes the input and emits a hidden state.
RecursiveLink transforms that hidden state into a compatible latent format.
Agent B receives the transformed state and continues reasoning.
The loop repeats for one or more recursion rounds.
A final decode step produces the visible text output.

This flow keeps the intermediate reasoning inside the browser and only surfaces the final answer when needed.

9. Why This Matters

The main value of this work is not simply that it runs locally. It is that it brings a richer coordination mechanism into a browser-native environment.

That matters for several reasons:

Privacy: data stays on-device.
Deployment simplicity: no backend orchestration is required.
Portability: users can run the system from a browser.
Research value: latent collaboration can be studied in a lightweight environment.
Decentralization: browser clients can potentially participate in distributed AI workflows.

RecursiveMAS WebLLM therefore sits at the intersection of browser AI, agent systems, and latent computation.

10. Limitations

This browser-native adaptation also has clear constraints:

Hidden-state manipulation is technically complex.
Browser memory and compute budgets are limited.
WebGPU performance varies by device and browser support.
Latent transfer can become unstable without careful normalization.

The system is a prototype and should not be treated as a full replacement for server-side training or large-scale agent orchestration.

These limitations are important to acknowledge because they define the realistic scope of the project.

11. Future Work

Several extensions are worth exploring next:

browser-to-browser latent communication,
dynamic agent graphs,
stronger RecursiveLink training strategies,
recursive memory modules,
evaluation across multiple browser/device classes.

A particularly interesting direction is to test whether browser-native latent recursion can preserve some of the efficiency benefits reported in the original RecursiveMAS paper when run on consumer hardware.

12. Project Context

This repository serves as a build pipeline for a latent-transfer-capable WebLLM model. It demonstrates how a compiled WebGPU model can expose last-layer hidden states and how a trained RecursiveLink can be assembled and consumed by a browser application.

Key implementation artifacts in this repo include:

expose_hidden.py — automated patcher for exposing hidden states in an MLC model definition.
build.sh — pipeline script for converting weights, generating config, and compiling a WebGPU runtime.
train_recursivelink.py — optional training script for RecursiveLink projection weights.

13. Conclusion

RecursiveMAS WebLLM is a browser-native exploration of RecursiveMAS-style latent collaboration. My work is based on RecursiveMAS (https://arxiv.org/abs/2604.25917) as the core idea, and adapts it into a WebGPU-backed runtime that runs entirely inside the browser.

The central idea is simple: if multi-agent reasoning can be expressed as latent-state recursion, then the browser may be able to host that process locally, privately, and without backend infrastructure. That makes the browser not just a user interface, but a viable execution layer for advanced agent research.

References

Recursive Multi-Agent Systems, arXiv:2604.25917 https://recursivemas.github.io/ Demo https://vishalmysore.github.io/recursiveMASDemo

Bringing Recursive Multi-Agent Systems to the Browser with WebLLM and WebGPU

vishalmysore — Mon, 22 Jun 2026 14:37:47 +0000

Most multi-agent AI systems have a hidden inefficiency.

Every time agents collaborate, they typically communicate by generating text, passing that text to another agent, and then re-processing it again. While this works, it's expensive, slow, and burns through tokens quickly.

What if agents could communicate without generating text at all?

That's the idea behind RecursiveMAS, a recent research framework that allows AI agents to collaborate directly through their internal latent representations instead of exchanging natural language.

Inspired by this research, I built recursiveMASWebLLM — a build pipeline that brings RecursiveMAS-style latent collaboration directly into the browser using WebLLM, MLC-LLM, and WebGPU.

The result is a fully client-side experimental platform for running recursive multi-agent systems on consumer hardware without requiring cloud GPUs.

The Problem with Traditional Multi-Agent Systems

Most agent frameworks operate like this:

Agent A → generates text
         ↓
Agent B → reads text and generates more text
         ↓
Agent C → reads text and generates final answer

Every handoff requires:

Token generation
Token transmission
Token re-processing

As the number of agents increases, the overhead grows rapidly.

A significant portion of the computation is spent translating thoughts into text and then converting that text back into internal representations.

This works, but it's not how neural networks naturally communicate.

What Is RecursiveMAS?

RecursiveMAS takes a different approach.

Instead of exchanging generated text, agents exchange their last-layer hidden states (latent representations).

Think of hidden states as the model's internal reasoning space before words are produced.

Agent A Hidden State
         ↓
RecursiveLink
         ↓
Agent B Hidden State
         ↓
RecursiveLink
         ↓
Agent C Hidden State

The entire multi-agent system becomes a recursive computation graph operating in latent space.

The original research introduces a lightweight component called RecursiveLink, which acts as a bridge between agents.

Rather than training or fine-tuning the underlying LLMs, only these small link modules are trained while the base models remain frozen.

This allows multiple agents to collaboratively refine reasoning before any text is generated.

Core Concepts

RecursiveLink

A lightweight residual network that transforms and transfers latent representations between agents.

Instead of passing:

"What is the answer?"

agents pass:

[hidden_state_vector]

This dramatically reduces communication overhead.

Inner Link

Allows an agent to recursively refine its own latent reasoning.

Agent
   ↓
Hidden State
   ↓
RecursiveLink
   ↓
Back Into Agent

This creates iterative self-improvement loops before decoding text.

Outer Link

Enables latent communication between different agents.

Agent A
   ↓
RecursiveLink
   ↓
Agent B

The research demonstrates that even heterogeneous models can participate in these recursive workflows.

System-Level Recursion

The entire multi-agent system can execute multiple refinement passes.

Pass 1
   ↓
Pass 2
   ↓
Pass 3
   ↓
Final Decode

Instead of generating intermediate text after every step, the system performs latent collaboration first and produces text only at the end.

Why This Matters

According to the RecursiveMAS research, latent-space collaboration delivers:

Higher benchmark accuracy
Reduced token consumption
Faster end-to-end inference
Better scalability across multiple agents

Reported results include:

Up to 75% reduction in token usage
1.2×–2.4× faster inference
Average accuracy improvements across reasoning, coding, science, and medical benchmarks

The key insight is that agents can collaborate more efficiently when communication occurs inside the neural representation space rather than through natural language.

The Challenge: Running RecursiveMAS in the Browser

The original RecursiveMAS implementation targets server environments and GPU inference stacks such as vLLM.

Browser-based AI introduces a major limitation:

WebLLM models do not normally expose internal hidden states.

Without access to hidden states, latent recursion is impossible.

That became the motivation for this project.

Introducing recursiveMASWebLLM

recursiveMASWebLLM is a specialized build pipeline for creating WebLLM models capable of latent-state transfer.

It extends the browser AI stack to expose the information required for RecursiveMAS-style recursion.

The goal is simple:

Research Paper
      ↓
Server GPU Implementation
      ↓
Browser-Compatible Runtime
      ↓
Accessible to Everyone

What This Project Adds

Hidden State Extraction

MLC-LLM is patched to expose:

get_last_hidden()

This allows browser applications to access last-layer hidden states directly during inference.

Without this capability, RecursiveMAS cannot function.

RecursiveLink Training Pipeline

The repository includes tooling to train and package RecursiveLinks.

train_recursivelink.py

Generated links are exported as:

recursivelink.json

These lightweight modules can then be loaded by browser-based agent systems.

Automated Browser Model Builds

The build pipeline supports:

Model conversion
Quantization
WebGPU compilation
WASM generation
Release packaging

Even small models can be built entirely through GitHub Actions without requiring local GPUs.

Browser Deployment

Outputs include:

.wasm
weights
recursivelink.json

These artifacts can be hosted on:

GitHub Releases
Hugging Face
Static web hosting

and loaded directly into browser applications.

Project Architecture

recursiveMASWebLLM
        │
        ▼
Build Pipeline
        │
        ▼
.wasm + weights + recursivelink.json
        │
        ▼
Hosted Artifacts
        │
        ▼
RecursiveMAS Playground
        │
        ▼
Browser-Based Recursive Agents

The builder generates everything needed for latent recursive collaboration in WebLLM-powered applications.

Why Browser-Based Recursive Agents Are Interesting

1. Democratizing Advanced AI Research

Researchers and developers can experiment with RecursiveMAS techniques without expensive cloud infrastructure.

If a device supports WebGPU, it can participate.

2. Interactive Experimentation

Developers can modify:

Recursion depth
Agent roles
Collaboration patterns
Prompt strategies

and immediately observe how latent collaboration affects outcomes.

3. Education

RecursiveMAS introduces a fundamentally different way of thinking about multi-agent systems.

Running it locally in a browser makes it easier to understand and teach.

4. Lower Latency

Reducing intermediate token generation is especially valuable in browser environments where responsiveness matters.

5. Future Extensions

Exposing hidden states opens the door to:

Latent planning systems
Browser-side distillation
Neural memory systems
Hybrid cloud/browser agents
Experimental reasoning architectures

RecursiveMAS is just one possible application.

Getting Started

Repository:

https://github.com/vishalmysore/recursiveMASWebLLM

The project includes:

Local build instructions
GitHub Actions workflows
RecursiveLink training utilities
Model packaging tools
Integration guidance for the RecursiveMAS playground

Looking Ahead

This project is still early, but it establishes the foundation for browser-native latent multi-agent systems.

Future work includes:

Larger model support
Improved model sharding
Additional collaboration patterns
Better WebGPU optimizations
Community-created RecursiveLinks
Integration with other browser AI frameworks

As browser AI continues to mature, I believe we'll see more experimentation move from cloud infrastructure to client-side environments.

RecursiveMAS demonstrates that some of the most interesting ideas in AI may not require massive server clusters—they may eventually run directly in the browser.

What do you think?

Could latent-space multi-agent systems become the next evolution of browser AI experimentation?

https://github.com/vishalmysore/recursiveMASWebLLM
https://recursivemas.github.io/
https://huggingface.co/VishalMysore/RecursiveMAS-0.5B-MLC

Stop Paying for Token APIs: How to Build a Serverless Multi-Agent Mesh in the Browser

vishalmysore — Wed, 17 Jun 2026 12:04:28 +0000

Every modern multi-agent architecture assumes a massive, expensive backend cloud infrastructure running hundreds of dollars in API token costs per hour. But what if an entire suite of specialized agents—Legal, Software, Security, Healthcare—could collaborate, negotiate, and execute complex tools completely localized inside consumer browser tabs, passing knowledge with zero intermediary servers?

Welcome to agentHerd—a radical paradigm shift in decentralized, sovereign artificial intelligence.

By combining the local client-side execution power of WebGPU with the direct, peer-to-peer networking capabilities of WebRTC, agentHerd turns ordinary browser tabs into highly scalable, self-hosting AI environments. Zero cloud costs. Zero central servers. Total data privacy.

Demo - https://vishalmysore.github.io/agentHerd/

The agentHerd Stack at a Glance

Inference: WebLLM / WebGPU (Running Llama 3, Phi-3, or Gemma natively in the browser).
Networking: Pure Serverless WebRTC Data Channels (Handshake via ephemeral URL hashes).
Data Isolation: Distributed Federated RAG (Knowledge discovery without centralized vector stores).
Determinism: Hybrid sandboxing (LLMs control personality/choice; sandboxed JavaScript handles immutable application rules).

1. The Core Paradox: Moving Inference and Networking to the Edge

Traditional AI agents are cloud-bound because of two heavy dependencies: compute (LLM inference) and orchestration (state management and messaging).

[Traditional Architecture]
Browser UI <---> Cloud Orchestrator <---> Vector DB <---> Expensive LLM APIs ($$$)

[agentHerd Architecture]
Browser Tab A (WebGPU + LLM) <======== WebRTC ========> Browser Tab B (WebGPU + LLM)
                                 (Direct P2P Link)

agentHerd breaks this centralization by pushing both layers entirely to the client device:

WebGPU for Compute: Instead of querying external endpoints, models are cached locally and executed on the client's GPU via WebGPU. The moment a user opens a tab, their device becomes an active AI compute node.
WebRTC for Orchestration: Instead of a centralized message broker routing agent dialogue, tabs establish direct, encrypted peer-to-peer WebRTC data channels.

Serverless Signaling via URL Hashes

WebRTC traditionally requires a signaling server to exchange Session Description Protocol (SDP) tokens. agentHerd implements an entirely serverless signaling option: the initial peer generates an SDP token encoded directly into a URL hash. Copying and sharing this URL establishes an absolute trust boundary—operating identically to an end-to-end encrypted chat room.

2. Federated Knowledge Retrieval (RAG) over WebRTC

One of the greatest challenges of collaborative AI is data sharing. Uploading private company manuals or personal codebases to a central cloud database poses severe security risks. agentHerd solves this via Federated Knowledge Retrieval.

[Peer A: Has "Legal_Doc.pdf"]             [Peer B: Needs Legal Context]
   |                                            |
   |-- 1. Generates Summary Card -------------->| (Broadcasts Summary to Mesh)
   |                                            |
   |                                            |-- 2. "I need data on Section 4"
   |<=- 3. Requests Chunk via WebRTC Data Channel-|
   |                                            |
   |-- 4. Runs Local RAG Engine                 |
   |-- 5. Extracts precise text snippet         |
   |                                            |
   |==- 6. Sends Answer Fragment via WebRTC ===>| (Received securely)

How the Flow Works:

Local Extraction: When a user uploads a document into their local agentHerd tab, the document never leaves their machine. The local browser model processes the file and generates a lightweight Summary Card.
Summary Broadcast: This abstract Summary Card is shared across the WebRTC mesh. Other peers know what knowledge Peer A possesses, but they do not have the raw data.
On-Demand Querying: When Peer B's agent requires deep granular details to answer a prompt, it queries Peer A over the direct WebRTC data channel.
Localized Verification: Peer A’s local RAG system searches its own memory space, extracts the specific matching snippet, passes it through its own WebGPU instance, and returns only the specific answer fragment to Peer B.

The Sovereign AI Rule: You hand the network your answers—never your data.

3. The Action Layer: Where Determinism Meets Personality

Generative models are inherently non-deterministic. If you ask an LLM to play chess against another LLM purely through text generation, the game will rapidly degrade into illegal moves and hallucinated board positions.

agentHerd solves this by splitting responsibilities through a strict separation of concerns:

The Persona (LLM): Manages choices, dialogue, strategic goals, and social banter.
The Guardrail (Deterministic Engine): An immutable, sandboxed environment (like a localized chess.js script) that enforces absolute operational rules.

When an agent wants to perform an action, it cannot arbitrary alter the state. It must output a structured JSON command envelope that is verified by every node in the mesh:

{
  "sender": "Agent_Alpha_Chess",
  "timestamp": 1718619837,
  "action": "EXECUTE_TOOL",
  "payload": {
    "tool_name": "CHESS_MOVE",
    "arguments": {
      "from": "e2",
      "to": "e4"
    }
  },
  "signature": "0x7f83b..."
}

If Agent_Alpha attempts to generate an illegal move, the deterministic script running on the peer nodes instantly rejects the packet, ensuring the integrity of the environment without requiring a central server to referee the state.

4. Operational Boundaries and Engineering Realities

Building completely within the constraints of a browser environment requires engineering trade-offs. Developers looking to leverage this stack should be aware of current boundaries:

VRAM and Model Swapping: Running models like Llama 3 (8B) or Phi-3 requires a modern GPU with sufficient VRAM. Attempting to open multiple heavy-inference browser tabs concurrently can saturate hardware resources. Multi-agent rooms perform best when using highly optimized 3B or smaller models optimized for web runtimes.
NAT Traversal & Corporate Firewalls: While public STUN servers successfully resolve connections for the majority of consumer network topologies, strict enterprise environments utilizing symmetric NATs often block direct WebRTC channels. In these scenarios, falling back to a dedicated, self-hosted TURN relay server becomes mandatory to handle the traffic.
Topology Scaling Limits: Because each browser tab must maintain WebRTC connections with other agents, a full-mesh topology (where every node connects to every other node) hits a browser-imposed performance wall as the group size scales. For massive clusters, the architecture transitions toward hybrid star/relay networks.

5. Join the Decentralized Frontier

The future of multi-agent collaboration isn't a massive corporate data center burning megawatts of energy to route your private API calls—it's the browser tab you already have open.

agentHerd proves that we can build highly complex, deeply collaborative, and perfectly private AI ecosystems using the open web standard tools already at our disposal.

The project is fully open-source and welcoming contributors. We are actively looking for developers to help build out new specialized agent domains, create native CLI-peer wrappers, and engineer custom tool integrations.

Explore the Codebase: agentHerd on GitHub
Launch a Room: Open the repository, generate your signaling hash, and invite your first agent herd today.
Contribute: Star the repo, open an issue, and let's build an unstoppable, serverless AI collective together.

Important distinction: agentHerd distributes cognition, not computation.
Each node runs its own model independently. The system does not combine GPUs to run larger models—it coordinates many smaller, autonomous agents working in parallel.

Foundation vs. Instruct vs. Chat Models: One Question, Three Answers

vishalmysore — Tue, 16 Jun 2026 23:08:32 +0000

A hands-on tutorial you can run for free in Google Colab.

Run it yourself: open foundation_instruct_chat_tutorial.ipynb in Google Colab and run every cell top to bottom. It uses the SmolLM2-135M family — small enough for a free CPU runtime, no GPU needed.

Why this confuses everyone

People say "LLM," "GPT," "an AI model," and "ChatGPT" as if they were the same thing. They aren't. There's a ladder of training stages between "a model that read the internet" and "an assistant you can chat with," and the words foundation, instruct, and chat mark the rungs.

The cleanest way to feel the difference is to do something deliberately unfair: ask the exact same question to three versions of the same model family and watch how differently they behave. Our question is deliberately boring so the behavior stands out:

"What is the capital of France?"

We use three checkpoints from Hugging Face's SmolLM2 family:

Model type	Hugging Face ID	One-line summary
Foundation (base)	`HuggingFaceTB/SmolLM2-135M`	Predicts the next token. Knows things, isn't helpful.
Instruct	`HuggingFaceTB/SmolLM2-135M-Instruct`	Fine-tuned to follow a single instruction.
Chat	`HuggingFaceTB/SmolLM2-135M-Instruct` (used conversationally)	Same weights, driven through a multi-turn message list.

Notice that the chat row reuses the instruct checkpoint. That's not a shortcut — it's the honest reality, and we'll come back to why.

1. The foundation model: a brilliant autocomplete

A foundation model (also called a base or pretrained model) is trained on exactly one objective: given a stretch of text, predict the next token. Nothing else. It reads a huge slice of the internet and gets very good at continuing text in a statistically plausible way.

What it is never taught is that a question deserves an answer. So when you feed it:

What is the capital of France?

it doesn't think "I should answer that." It thinks "On the internet, what usually **comes after* a line like this?"* And the answer is often… more quiz questions, a worksheet, or a tangent:

What is the capital of France? What is the capital of Germany? What is the
capital of Italy? ...

In the notebook we pass the raw string straight into the pipeline with no formatting:

base_pipe = pipeline("text-generation", model="HuggingFaceTB/SmolLM2-135M")
base_raw_out = base_pipe(test_query, max_new_tokens=30, do_sample=False)
print(base_raw_out[0]['generated_text'])

Takeaway: a foundation model is a text completer, not an assistant. It contains enormous knowledge but has no concept of being helpful. It's the raw clay everything else is shaped from.

2. The instruct model: teaching the model to answer

An instruct model starts from that same base model and goes through a second stage of training — fine-tuning on (instruction → response) pairs. Thousands to millions of examples of the shape "Here's a request. Here's a good response." This teaches the model a new contract: when the user asks for something, actually do it and then stop.

But there's a crucial detail people miss: an instruct model only behaves correctly when you wrap your text in the exact special format it was trained on. That format uses control tokens — for SmolLM2 they look like this:

<|im_start|>user
What is the capital of France?<|im_end|>
<|im_start|>assistant

You don't type those tokens by hand. Every instruct model ships with a chat template baked into its tokenizer that builds them for you:

tokenizer = AutoTokenizer.from_pretrained(instruct_id)
formatted_prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": test_query}],
    tokenize=False,
    add_generation_prompt=True,  # appends the 'assistant' cue
)

Feed that to the same-sized model and you get a clean, direct answer:

The capital of France is Paris.

The notebook prints the formatted prompt before generating, so you can literally see the hidden scaffolding the model receives. That "aha" — oh, there's a whole structure under the hood — is the most important thing in the tutorial.

Takeaway: an instruct model = a base model + instruction tuning + a required prompt format. Skip the format and even a well-trained instruct model can fall back to rambling.

3. The chat model: memory across turns

Here's the part that surprises people: a chat model is usually the same weights as the instruct model. The difference isn't what the model is — it's how you drive it.

Instead of one instruction in, one response out, you maintain a running list of role-tagged messages:

chat_history = [
    {"role": "user", "content": "What is the capital of France?"},
]
chat_out = chat_pipe(chat_history, max_new_tokens=30)

The pipeline applies the chat template for you and returns the whole conversation with the assistant's reply appended. For a single turn, that looks identical to the instruct example. The magic only appears when the conversation continues.

So in the notebook we append the reply and ask a deliberately vague follow-up:

conversation = chat_out[0]['generated_text']        # user + assistant so far
conversation.append({"role": "user",
                     "content": "And what is a famous landmark there?"})
follow_up = chat_pipe(conversation, max_new_tokens=40)

The word "there" is meaningless on its own. But because we passed the entire history, the model resolves "there" → Paris and names a landmark. That carried-over context is what turns a one-shot Q&A into something that feels like a conversation.

Takeaway: a chat model is an instruct model driven through a multi-turn message list, so each new turn can use the previous turns as context. The system prompt, the user/assistant roles, and the growing history are the "chat" part.

The whole picture in one table

Model	Trained to…	You give it…	Reply to "What is the capital of France?"
Foundation	continue text	a raw string	echoes / continues the document — may never answer
Instruct	follow one instruction	a chat-templated string	a direct answer: "The capital of France is Paris."
Chat	converse over many turns	a list of messages	a direct answer + remembers context for follow-ups

Read top to bottom, it's a progression, not three unrelated things:

Foundation learns the world by predicting text.
Instruct fine-tunes that knowledge into do-what-I-ask behavior — and demands a specific prompt format.
Chat wraps the instruct model in a multi-turn interface so context flows across turns.

When you talk to a commercial assistant, you're using stage 3, sitting on stage 2, built on stage 1.

A note on honesty and scale

SmolLM2-135M is tiny — about 135 million parameters, versus the tens or hundreds of billions in frontier models. At this size the model will sometimes get a fact wrong, repeat itself, or trail off. That's expected, and it's not the point. The tutorial is designed to make the behavioral gap between the three modes visible on a free laptop or Colab CPU — not to win a trivia contest. The exact same three-stage structure scales all the way up to the largest models in production.

Run it and tinker

Open foundation_instruct_chat_tutorial.ipynb in Google Colab (File → Open notebook → Upload, or push it to GitHub and use the Colab badge).
Run all cells (Runtime → Run all). The first run downloads the models — give it a minute.
Experiment:
- Change test_query to something open-ended like "Write a haiku about the sea." and watch how the three modes diverge even more.
- Set do_sample=True with temperature=0.7 for more varied, creative output.
- Swap in a larger sibling such as HuggingFaceTB/SmolLM2-360M-Instruct and feel the quality jump.

Once you've seen the three behaviors with your own eyes, the vocabulary — base, instruct, chat, chat template, system prompt — stops being jargon and starts being obvious.

Happy experimenting! 🚀

DEV Community: vishalmysore

PROOF for AI Agents: A Technical Scoring Rubric For Self Evaluation

The Rubric, Formalized

Healthcare: Clinical Intake and Triage Support

Manufacturing: Production-Line Anomaly Response

What the Two Domains Have in Common

Is This a Real AI Agent? Ask for PROOF

Why "Agent" Stopped Meaning Anything

The PROOF Framework

Scoring It

What to Actually Ask in the Room

Building Local AI Agents in Java with Tools4AI and Ollama: An Insurance Claims Use Case

Table of Contents

Why local AI agents matter for insurance

What is Tools4AI?

Why Ollama + Java is a great fit for regulated data

Prerequisites

Step 1 — Install and run Ollama

Step 2 — Add the Tools4AI dependency

Step 3 — Point Tools4AI at Ollama

Step 4 — Your first local AI call

The insurance use case: an FNOL claims triage agent

1. Define your business actions as agents

The Real Reason 40% of Agentic AI Projects Are Getting Cancelled (It's Not the Model)

Pilots only need one layer to hold. Production needs all four.

1. The Spec Layer — why the agent is "almost right"

2. The Guardrail Layer — why "it worked in staging" doesn't mean it's safe

3. The Memory Layer — why the agent keeps making the same mistake

4. The Coordination Layer — why the failure isn't in any single agent

Why this explains the 40%

Serverless AI in a Browser Tab: Java WebAssembly + Local WebGPU LLMs

A deep technical whitepaper on building a zero-infrastructure RAG architecture where the business logic is Java compiled to WebAssembly and the intelligence is a quantized LLM running on your own GPU

Abstract

Project Disclaimer & Intent

1. The problem with the current architecture

2. What is WebAssembly?

2.1 WasmGC: the part that makes Java practical

3. Why Wasm is a game changer

4. Writing the business core in Java

4.1 Separation of concerns

4.2 The interop surface

4.3 A hard-won lesson: the synchronous boundary

5. The serverless architecture

6. Local intelligence: WebGPU, on-device embeddings, and a browser LLM

6.1 WebGPU

6.2 WebLLM + a quantized SLM

6.3 On-device embeddings

7. The combined architecture: serverless RAG

7.1 Why the Java core is genuine business logic, not a wrapper

8. Analysis

8.1 Privacy

8.2 Cost

8.3 Performance & honest trade-offs

9. When to reach for this architecture

10. Future directions

11. Conclusion

Appendix: reference stack

webSLM: Fine-tuning, Compiling, and Running Domain-Specific Small Language Models Entirely in the Browser

1. Introduction

1.1 Design principles

2. Background

2.1 SLM vs. quantized LLM

2.2 Fine-tuned SLM vs. RAG

2.3 The runtime stack: WebLLM + MLC-LLM

3. System architecture

4. Stage 1 — Fine-tuning on Colab (LoRA SFT)

4.1 Data format

4.2 The training recipe

4.3 Running it on Colab

5. Stage 2 — Compiling to WebGPU via GitHub Actions

5.1 Why build the toolchain from source

5.2 The MLC compile pipeline

5.2.1 The config-normalization gotcha

5.2.2 Quantization choice

5.3 Inputs and outputs

6. Stage 3 — Running in the browser with WebLLM

6.1 Registering a custom model — appConfig goes in the constructor

6.2 model must be a full URL

6.3 Pin the runtime to the wasm's ABI

6.4 Inference and decoding

6.1 Registering a custom model — `appConfig` goes in the constructor

6.2 `model` must be a full URL

4.2.1 Main Entry Point (`main.js`)

4.2.2 Latent Forward (`latent-chain.js`)