HeadLess BAI

Akshit Sharma — Sat, 13 Jun 2026 11:45:25 +0000

I Compiled a Custom Chromium Binary to Make AI Fix Bad UIs — Here's How It Works

Most UI optimization tools work the same way: inject some JavaScript, watch click events, run A/B tests, wait months for results.

I wanted something different. Something that could see the DOM the way the browser actually sees it — at the geometry level — and use that to drive real layout decisions without guesswork.

So I built Headless BAI: a behavioral AI pipeline that compiles a patched Chromium binary, intercepts DOM bounding box data in C++, clusters user behavior with K-Means, and routes layout patches through a Llama-3-8B LLM. The result: 31.42% reduction in UI spatial friction across 1,000+ user sessions.

Here's exactly how it works, layer by layer.

The Problem With JavaScript-Based DOM Inspection

Every existing approach — Puppeteer, Playwright, even Chrome DevTools Protocol — has the same limitation: they're sitting outside the rendering pipeline. When you call getBoundingClientRect() in JS, you're asking the browser to compute and serialize layout data across a process boundary. That costs time, introduces CLS artifacts, and you still don't get the raw geometry the compositor is working with.

I wanted native access. Which meant one thing: patch Chromium itself.

Layer 1: Custom Chromium Binary with C++ DOM Instrumentation

The chromium-patches/ folder contains the core of this — diffs applied directly to Chromium's rendering pipeline.

The patch hooks into the layout engine at the point where bounding boxes are finalized for compositing. Instead of waiting for JS to request this data, we intercept it at the C++ level and serialize it to a shared memory buffer. The result: sub-100ms DOM geometry rehydration with zero CLS (Cumulative Layout Shift).

The compiled binary lives in out/Default/ — a full headless Chromium build with our instrumentation baked in.

This is the layer most people skip. It's painful to set up (Chromium's build system is not forgiving), but it's what makes everything downstream actually meaningful.

Layer 2: sensor.js → FastAPI → Supabase Telemetry Pipeline

Once you have clean geometry data, you need behavioral signal on top of it.

sensor.js is a zero-dependency vanilla JS tracker that captures:

Mouse movement trajectories
Scroll depth and velocity
Click coordinates relative to element bounding boxes
Hover dwell time per element region

Every event gets timestamped and batched. The FastAPI backend (bai-final-year/backend/) ingests these, validates them against our bot filter, and writes to Supabase with Redis caching in front for burst tolerance.

In practice: 10,800+ behavioral events processed across the test sessions, in real time.

Layer 3: Behavioral Cohort Discovery via K-Means Clustering

Raw events are noisy. Before you can do anything useful, you need to segment users by how they actually move through the UI.

The mass_evaluation/ module runs nightly batch jobs that:

Featurize sessions: friction score per zone, scroll abandonment depth, click dispersion radius
Run K-Means with WCSS optimization (final WCSS: 23.24, 3 cohorts)
Apply EMA weighting to smooth temporal drift in behavior patterns

Three cohorts emerged consistently across sessions: high-friction explorers, direct navigators, and scroll-heavy skimmers. Each one needs a different layout strategy.

Layer 4: Llama-3-8B LLM Layout Routing (T=0.1)

Here's where it gets interesting.

Once cohorts are identified, we don't hardcode layout rules. Instead, we prompt Llama-3-8B with the cluster's behavioral fingerprint and the current DOM geometry, and ask it to generate a JSON layout patch.

Temperature is set to 0.1 — near-deterministic. This is intentional. We're not using the LLM for creativity; we're using it as a structured reasoning engine that can translate "cohort 2 users consistently abandon at the 340px scroll depth near the CTA block" into a concrete spatial adjustment.

Output format:

{
  "target_element": "#hero-cta",
  "adjustment": "translate_y",
  "delta_px": -48,
  "rationale": "High scroll-abandonment cohort loses CTA visibility before natural pause point"
}

Patch generation runs in under 500ms per session cohort.

The Metric: 1D Kinematic Friction Score

Standard UX metrics (bounce rate, time-on-page) are too coarse for layout-level optimization. I needed something that could quantify spatial friction — how much a layout fights the user's natural movement patterns.

The Kinematic Friction Score models user cursor/scroll behavior as a 1D kinematic system. High friction = layout elements that interrupt natural movement trajectories. Low friction = elements aligned with how users are actually moving.

After applying AI-generated layout patches across 1,000+ sessions:

Session-weighted friction reduction: 31.42%

That's not a vanity metric. It's computed per-session, weighted by engagement depth, and aggregated across all three behavioral cohorts.

Tech Stack Summary

Layer	Tech
DOM Extraction	C++ (Chromium internals, custom patches)
Event Ingestion	FastAPI + Redis + Supabase
Frontend Tracking	Vanilla JS (zero dependencies)
Clustering	scikit-learn K-Means + EMA weighting
LLM Routing	Llama-3-8B via Groq (T=0.1)
Infrastructure	Docker Compose

What I'd Do Differently

Bot filtering is harder than it looks. Our current filter uses statistical heuristics (event timing distributions, movement entropy). A dedicated ML classifier here would be more robust.

The Chromium build pipeline is brutal. 8+ hour compile times on a standard machine. If I were doing this again, I'd look at intercepting at the DevTools Protocol level first to validate the concept before going full custom binary.

Cohort stability across sessions needs work. K-Means doesn't guarantee stable cluster assignments between runs. Moving to an online clustering approach (something like BIRCH or a streaming K-Means) would make the behavioral model genuinely continuous rather than batch-nightly.

Code

→ github.com/Aksharma127/Headless_BAI_orignal

The repo includes the Chromium patches, full backend, clustering pipeline, and evaluation outputs. Still under active development — the architecture is solid but there's cleanup to do.

If you've worked on browser internals, behavioral analytics, or LLM-driven UI systems, I'd genuinely like to hear what you think.

machinelearning #webdev #cpp #ai

Everyone is using LLMs wrong........

Akshit Sharma — Sat, 09 May 2026 19:19:18 +0000

---

Can an LLM "See" a Room Just by Listening?

Everyone is using LLMs wrong. We are obsessed with Speech-to-Text. We take audio, flatten it into text, and feed it to a chatbot.

But what happens if you bypass the text entirely?

What if you feed the raw acoustic tensor data directly into a native multimodal LLM and ask it to connect the physical dots?

I am running an experiment to see if an AI can blindly "see" a room just by listening to it.

The Experimental Setup

I am blasting a 3kHz–6kHz broadband frequency chirp (a mathematical "bell ping") inside a highly reverberant, non-rectangular tile box.

The Bounds: Tight. X=104", Y=101", Z=97".
The Geometry: It is a geometric nightmare. There’s a 21.5-inch inward offset on one wall acting as a diffuser.
The Obstacles: A urinal sitting at coordinate (0, 45, 32) and a toilet at (0, 76, 0).
The Target: My primary test target is located precisely at (0, 23, 15).

The Traditional Approach

Normally, calculating this requires hardcore Synthetic Aperture Sonar (SAS) math. You sweep a directional emitter, capture the sound from multiple microphone edges (M1, M2, etc.), and calculate the Time Difference of Arrival (TDOA).

For my target object, the acoustic shadow hits at exactly ~1.92 ms.

The "Crazy" Idea

Instead of writing a manual C++ or Julia script to subtract the empty room baseline ($\Delta R(t)$) and plot intersecting hyperbolas, what if we just feed the raw, overlapping multipath reflection audio directly into an LLM as native tokens?

No text. No pre-processing. Just raw environmental interaction data.

If the model can understand the hidden sequential context of a billion text parameters, can it implicitly learn the physics of sound propagation? Can it identify that the localized energy redistribution at 1.92 ms is an 8-inch object, while the dense tail after 10 ms is just the tile ceiling?

Can an LLM act as a biological auditory cortex and map a 3D coordinate system entirely in the dark?

DEV Community: Akshit Sharma