Building an On-Device Q&A Agent with Chrome Built-in AI and the Prompt API

Running large language models typically involves cloud API endpoints, paying per token, managing rate limits, and transmitting sensitive user queries across the network.

With Chrome's Built-in AI initiatives (via the WICG Prompt API and Gemini Nano), browsers can now execute small language models (SLMs) directly on the user's hardware. Inference happens entirely on-device: zero API keys to expose in frontend bundles, zero cloud infrastructure bills, zero network latency after weight loading, and complete client-side data privacy.

This article walks through the technical mechanics of integrating Chrome's Prompt API into a modern JavaScript/TypeScript application, including capability detection, session lifecycle, streaming token consumption, in-context grounding, and deterministic parameter tuning.

---

1. The Prompt API Architecture

Chrome exposes on-device capabilities through the window.ai namespace (and globally as LanguageModel). Under the hood, Chromium interfaces with an optimized on-device foundation model (Gemini Nano) managed by the browser's Optimization Guide component.

The API contract consists of three primary phases:

1. Availability Assessment: Probing whether hardware and browser flags support on-device execution.
2. Session Creation & Model Download: Initializing a session with optional system instructions and monitoring download progress if weights are not yet cached.
3. Execution & Token Streaming: Dispatching prompts via single-shot promises or asynchronous streams, followed by explicit resource teardown.

---

2. Feature Detection and Availability Checking

Before invoking any inference methods, you must verify that the browser supports the API and determine whether model weights are already loaded or require downloading.

export type AIAvailabilityStatus =
  | 'readily'
  | 'available'
  | 'after-download'
  | 'downloadable'
  | 'downloading'
  | 'unavailable'
  | 'no'
  | 'unsupported';

export function getLanguageModelAPI() {
  if (typeof window === 'undefined') return null;

  const anyWin = window as unknown as {
    ai?: { languageModel?: any; assistant?: any };
    LanguageModel?: any;
  };

  return (
    anyWin.ai?.languageModel ||
    anyWin.ai?.assistant ||
    anyWin.LanguageModel ||
    null
  );
}

export async function checkAIAvailability(): Promise<AIAvailabilityStatus> {
  const api = getLanguageModelAPI();
  if (!api) return 'unsupported';

  if (typeof api.availability === 'function') {
    const status = await api.availability();
    return status || 'unsupported';
  }

  if (typeof api.capabilities === 'function') {
    const caps = await api.capabilities();
    return caps?.available || 'unsupported';
  }

  return 'unsupported';
}

The status values dictate UI state:

• readily / available: Model weights exist in memory/disk cache; execution starts immediately.
• after-download / downloadable: Model execution is supported, but Chrome must fetch the weights (~1.5GB - 2.5GB). You should track and render download progress.
• no / unsupported: The host machine does not satisfy minimum hardware requirements (GPU/VRAM) or the browser flags are disabled.

---

3. Session Initialization and Weight Download Monitoring

When calling create(), you can pass configuration options:

• systemPrompt: High-level persona and constraints.
• temperature: Sampling temperature (0.0 to 1.0). Lower values yield deterministic, factual output.
• topK: Number of highest-probability tokens considered.
• monitor: An event listener callback tracking download bytes.
• signal: An AbortSignal to cancel session instantiation.

const api = getLanguageModelAPI();

const session = await api.create({
  systemPrompt: 'You are a technical assistant specializing in software engineering telemetry.',
  temperature: 0.1,
  topK: 1,
  signal: abortController.signal,
  monitor: (monitorTarget: EventTarget) => {
    monitorTarget.addEventListener('downloadprogress', (event: Event) => {
      const prog = event as ProgressEvent;
      if (prog.total > 0) {
        const percentage = Math.round((prog.loaded / prog.total) * 100);
        console.log(`Downloading model weights: ${percentage}%`);
      }
    });
  },
});

---

4. Grounding Small On-Device Models (In-Context Prompting)

Gemini Nano is a lightweight model (~3 billion parameters). Unlike cloud models with massive parameter counts (such as Gemini 1.5 Pro or GPT-4), on-device SLMs have narrower context windows and tend to suffer from attention attenuation if domain context is provided strictly as a detached system prompt.

If you pass domain knowledge solely in systemPrompt during create(), the model may drift or produce generic answers when the conversation advances.

To achieve strict factual fidelity (e.g., answering questions exclusively from a structured career timeline or technical dataset), format the verified knowledge directly into the prompt envelope at query time:

export function buildGroundedPrompt(
  facts: string,
  userQuery: string,
  history: { role: string; content: string }[] = []
): string {
  let conversationBlock = '';
  if (history.length > 0) {
    const recent = history.slice(-3);
    conversationBlock = `RECENT CONVERSATION:\n${recent
      .map((m) => `${m.role === 'user' ? 'User' : 'Assistant'}: ${m.content}`)
      .join('\n')}\n\n`;
  }

  return `You are a verified technical assistant. Answer the user question accurately, concisely, and factually using ONLY the verified facts below. If the answer is not contained in the facts, explicitly state that the information is unavailable.

VERIFIED FACTS:
${facts}

${conversationBlock}USER QUESTION: ${userQuery}

CONCISE FACTUAL ANSWER:`;
}

Deterministic Parameter Tuning

For retrieval and Q&A tasks where hallucinations must be avoided:

• Set temperature: 0.1 or 0.0.
• Set topK: 1.

This forces the model to select the highest-confidence token at each step, preventing speculative drift.

---

5. Streaming Token Consumption and Lifecycle Management

To avoid blocking UI rendering while generating output, consume the streaming interface (promptStreaming). Depending on the exact Chromium version and specification draft, promptStreaming returns an AsyncIterable<string> or a ReadableStream<string>.

Here is a normalization handler that supports both paradigms and cleans up the session after execution:

export async function executePromptStreaming({
  session,
  prompt,
  onChunk,
  signal,
}: {
  session: any;
  prompt: string;
  onChunk: (accumulated: string, delta: string) => void;
  signal?: AbortSignal;
}): Promise<string> {
  if (typeof session.promptStreaming === 'function') {
    const streamResult = session.promptStreaming(prompt, { signal });
    let accumulated = '';

    // Case 1: AsyncIterable
    if (
      streamResult &&
      typeof streamResult[Symbol.asyncIterator] === 'function'
    ) {
      for await (const chunk of streamResult) {
        if (signal?.aborted) break;
        accumulated += chunk;
        onChunk(accumulated, chunk);
      }
      return accumulated;
    }

    // Case 2: ReadableStream
    if (
      streamResult &&
      typeof streamResult.getReader === 'function'
    ) {
      const reader = streamResult.getReader();
      try {
        while (true) {
          if (signal?.aborted) break;
          const { done, value } = await reader.read();
          if (done) break;
          if (value) {
            accumulated += value;
            onChunk(accumulated, value);
          }
        }
      } finally {
        reader.releaseLock();
      }
      return accumulated;
    }
  }

  // Fallback: Non-streaming execution
  const singleShotResult = await session.prompt(prompt, { signal });
  onChunk(singleShotResult, singleShotResult);
  return singleShotResult;
}

Critical: Resource Disposal

Each active session retains model state and memory allocations in the browser process. Always invoke session.destroy() inside a finally block or when unmounting React components to prevent memory leaks:

try {
  const result = await executePromptStreaming({ session, prompt, onChunk, signal });
  return result;
} finally {
  try {
    session.destroy();
  } catch (err) {
    // Suppress teardown errors
  }
}

---

6. React Integration Pattern

In a React or Next.js SPA, wrap the session logic in custom hooks or state handlers. Below is an example pattern maintaining conversation state and abort signals:

import React, { useState, useRef, useEffect } from 'react';
import {
  checkAIAvailability,
  getLanguageModelAPI,
  buildGroundedPrompt,
  executePromptStreaming,
} from './aiAssistant';

export const OnDeviceChat: React.FC<{ datasetContext: string }> = ({ datasetContext }) => {
  const [messages, setMessages] = useState<{ role: string; content: string }[]>([]);
  const [input, setInput] = useState('');
  const [isGenerating, setIsGenerating] = useState(false);
  const [downloadProgress, setDownloadProgress] = useState<number | null>(null);
  const abortControllerRef = useRef<AbortController | null>(null);

  const handleSubmit = async (e: React.FormEvent) => {
    e.preventDefault();
    if (!input.trim() || isGenerating) return;

    const userText = input.trim();
    setInput('');
    setIsGenerating(true);

    const abortController = new AbortController();
    abortControllerRef.current = abortController;

    const newMessages = [
      ...messages,
      { role: 'user', content: userText },
      { role: 'assistant', content: '' },
    ];
    setMessages(newMessages);

    try {
      const api = getLanguageModelAPI();
      if (!api) throw new Error('Prompt API not supported on this browser.');

      const session = await api.create({
        systemPrompt: 'You are a factual Q&A assistant.',
        temperature: 0.1,
        topK: 1,
        signal: abortController.signal,
        monitor: (m: EventTarget) => {
          m.addEventListener('downloadprogress', (e: any) => {
            if (e.total) setDownloadProgress(Math.round((e.loaded / e.total) * 100));
          });
        },
      });

      try {
        const fullPrompt = buildGroundedPrompt(datasetContext, userText, messages);

        await executePromptStreaming({
          session,
          prompt: fullPrompt,
          signal: abortController.signal,
          onChunk: (accumulated) => {
            setMessages((prev) => {
              const updated = [...prev];
              updated[updated.length - 1] = { role: 'assistant', content: accumulated };
              return updated;
            });
          },
        });
      } finally {
        session.destroy();
      }
    } catch (err: any) {
      if (!abortController.signal.aborted) {
        console.error('Inference error:', err);
      }
    } finally {
      setIsGenerating(false);
      setDownloadProgress(null);
    }
  };

  const handleAbort = () => {
    abortControllerRef.current?.abort();
    setIsGenerating(false);
  };

  return (
    <div>
      {downloadProgress !== null && (
        <div>Downloading on-device weights: {downloadProgress}%</div>
      )}

      <div>
        {messages.map((m, idx) => (
          <div key={idx} className={m.role}>
            <strong>{m.role}:</strong> {m.content}
          </div>
        ))}
      </div>

      <form onSubmit={handleSubmit}>
        <input
          value={input}
          onChange={(e) => setInput(e.target.value)}
          placeholder="Ask a question..."
          disabled={isGenerating}
        />
        {isGenerating ? (
          <button type="button" onClick={handleAbort}>Stop</button>
        ) : (
          <button type="submit">Submit</button>
        )}
      </form>
    </div>
  );
};

---

7. How to Enable the Prompt API in Chrome

Because the WICG Prompt API is rolling out across Chromium channels, testing locally requires enabling experimental flags:

1. Open Google Chrome (version 128+ or Chrome Canary).
2. Go to chrome://flags/#prompt-api-for-gemini-nano and set to Enabled.
3. Go to chrome://flags/#optimization-guide-on-device-model and set to Enabled BypassPerfRequirement.
4. Relaunch Chrome.
5. Visit chrome://components and find Optimization Guide On Device Model. Click Check for update to ensure weights are fully fetched.

---

Summary

Chrome's Built-in AI and Prompt API represent a fundamental shift in client-side web capabilities. By moving baseline reasoning, extraction, and Q&A workloads directly onto client hardware:

• Network egress costs and backend server bills drop to zero.
• User data never leaves the client boundary.
• Web apps can provide offline and zero-latency interactive features.

By pairing small on-device models with strict in-context grounding and low-temperature sampling, you can build production-ready, highly reliable Q&A agents directly in the browser.