DEV Community: Navjyot N

Before You Claim AI Made Your Team Faster, Measure It

Navjyot N — Mon, 17 Aug 2026 03:04:12 +0000

Every few months, I find myself in some version of the same conversation.

A team has been using AI coding tools for a while. The excitement has cooled into a more practical question: what did we actually get from this?

It is a fair question. Leaders have bought licenses, teams have changed habits, and somebody eventually has to explain whether any of it helped. The awkward part is that the usual answers do not feel very satisfying. Seat counts say who had access. Surveys say how people felt. Vendor dashboards say a tool was opened.

None of that tells you what happened in the codebase.

That is why I built whodunit. It is a small open-source tool for answering AI adoption questions from evidence that already exists: git commits, local agent logs, and, if you want the shared view, delivery data from DevLake.

The goal is not to make AI look good. It is to make the claims harder to misuse.

The questions people actually ask

These are the questions I keep hearing from clients and engineering leaders.

Is my team actually adopting AI, and is adoption spread evenly?
Can you show me the productivity improvement?
Are people using AI well, or just chatting with it?
Is AI helping around the edges, or writing most of the code?
How much AI-written code is accepted?
Where is AI being used: features, fixes, tests, docs?
How much autonomy are developers giving the agent?
Can we see churn for AI-written code?
Which models are being used most?
Did PR cycle time improve?
How long did adoption take?

I like this list because it is honest. It is not a polished strategy deck. It is what people ask when they are trying to make sense of a real investment.

Some of these questions are answerable. Some are partly answerable. One of them, the productivity percentage, is the one I am most careful with.

The small thing whodunit adds

whodunit adds a plain git trailer to commits:

AI-Attribution: v=1; status=assisted; method=intersected; agent=claude-code; agent_version=2.1.228; ratio=0.62; model=claude-opus-5; session=a3f9e21c

There is no magic in the format. That is the point. It is just git metadata, readable by any tool that can read a commit message.

The trailer records whether a commit was AI-assisted, which agent was involved, which model was reported, and how strong the evidence is. The strongest method, intersected, means text the agent produced survived into what was staged.

When the evidence is missing, whodunit records that too. It uses undetermined instead of quietly deciding "no AI was used." That distinction between absence and zero is the rule the whole tool is built on. That distinction matters more than it sounds. A missing signal is not a zero. It is just missing.

The one-screen answer

The executive dashboard is the page for the person who will not open the rest. It shows assisted commits, coverage, acceptance rate, active sessions, and delivery comparison in one place.

The most important number on that page is not the flashiest one. It is coverage.

Coverage tells you how much of the commit history has a valid attribution trailer. Adoption tells you how much of that covered history was assisted. Those are different claims, and mixing them up is where bad AI metrics start.

A team can have high coverage and low adoption. That is a valid finding. It means the instrumentation is working and the evidence says AI is not showing up much in committed work.

A team can also have low coverage and a beautiful adoption number. That one should make you pause.

Adoption is a spread, not a slogan

When someone asks "is the team adopting this?", the team average is not enough.

I want to know whether adoption is spread across the group or carried by two people. I want to know who has no assisted commits at all. I want to know whether the data covers enough of the repo to trust the answer.

That is why the adoption view shows contributor spread beside coverage. The contributor identity comes from the git committer email that is already present in commits. whodunit is not watching developers. It is making existing commit metadata easier to read.

There is one catch worth saying gently but clearly: the clock starts when instrumentation starts.

If your team used AI for six months before installing hooks, those six months are not magically recoverable. whodunit can tell you what happened after the trailers started landing. For a real before-and-after story, capture a baseline before the rollout with dun baseline capture.

Miss that window and you can still learn a lot. You just have to be honest about where the measurement begins.

"Are they using it well?" is only partly measurable

This is the question that can get uncomfortable fast.

People ask whether developers know compaction, whether they choose the right model, whether they use skills and sub-agents, whether they turn everyday work into good AI workflows. I understand why they ask. Training budgets depend on it.

But I do not think a dashboard should pretend to read competence from a transcript.

What whodunit can show is session shape. Did a session only involve conversation? Did it call tools? Did it edit files? Did it use many tools or MCP calls? Which models were used? Did the session compact?

That gives you a useful proxy for "just chatting" versus "using the agent to do work." It does not tell you whether a developer is skilled with AI. That still needs human context: pairing, coaching, code review, and conversation.

I would rather leave that boundary visible than build a confidence score no one should trust.

Where AI shows up in the work

For questions like "is AI being used for features, tests, docs, or fixes?", whodunit leans on commit prefixes and path heuristics.

That is useful, but only if your repository gives the tool something decent to work with. If your team uses Conventional Commits, a feat: commit is a pretty good signal. If your team does not, the feature panel will look empty even when people are shipping features all week.

So I treat purpose as a label, not an observation.

This is a recurring theme in the project: the chart should say what it knows, and just as importantly, what it does not know.

Autonomy changes the shape of a session

Autonomy was more interesting than I expected.

Different tools have different permission vocabularies. Codex may report modes like never and on-request. Claude Code may report acceptEdits, default, or auto. whodunit keeps those names instead of pretending they all map neatly onto the same ladder.

The useful question is not only "how often did people grant autonomy?" It is "what happened once they did?"

On my data, high-autonomy sessions were fewer, but much denser. Tool calls per session captured that better than raw session count. Still, I would be careful with the interpretation. A mode that allows more actions will naturally produce more actions. Tool calls are activity, not delivered value.

The productivity number everyone wants

This is the question that started the whole thing:

Show me the productivity improvement after AI adoption.

I wish that number were easier to produce honestly. It would make the meeting shorter.

The problem is that assisted and unassisted work are not random groups. Developers choose when to use an agent. They may use it for larger changes, repetitive migrations, unfamiliar code, tests, or work they already know how to delegate.

So if assisted commits are larger, what does that prove?

Maybe AI helped people take on bigger work. Maybe it encouraged bigger diffs. Maybe people simply used it on a different kind of task. The same chart supports all three stories.

That is why whodunit shows comparisons, not productivity gains. It can show change size, churn, acceptance, purpose, adoption, and cycle-time differences when delivery data is wired up. It should not turn those into a percentage and call it ROI.

A baseline helps. If you capture a pre-adoption window before installing hooks, dun delta can compare that period with a later one. That is much stronger than comparing assisted and unassisted commits inside the same period.

It is still observational. Teams change, codebases change, projects change. The baseline gives you a better conversation, not a laboratory experiment.

That may sound unsatisfying. In practice, I think it is a relief. It lets you say, "Here is what moved, here is what did not, and here is what we cannot honestly claim."

The mistakes I am trying to avoid

The biggest metric mistakes are usually flattering.

Empty is not zero. If an agent does not report token usage, showing zero makes it look free. If a commit has no attribution evidence, treating it as "not AI" makes adoption look cleaner than the evidence allows.

Tokens are not dollars. A transcript does not know whether someone is using a subscription, an enterprise contract, or API billing. whodunit reports tokens and cache behavior. It does not invent a price.

Cache writes count. Cache read ratio should include uncached input, cache writes, and cache reads. Leaving writes out made one of my early analyses look much better than it was.

A fix rate is not a defect rate. Fix-labelled commits are a rework proxy. They are useful, but they do not prove AI caused or prevented defects.

A difference is not a gain. This is the one I keep coming back to. If the groups are self-selected, the comparison can be useful without being causal.

None of this makes the tool less useful. It makes it safer to use.

How it is built

There are two pieces.

At a high level, whodunit keeps the sensitive work local. The developer machine reads git and local agent logs, stamps attribution into commits, and keeps a local journal. A team only gets the shared dashboards if someone chooses to sync the whodunit_* tables into the optional DevLake layer.

The first is dun, a Go CLI that runs locally:

cd your-repo
dun init          # installs the git hooks
git commit ...    # the trailer is stamped automatically
dun status        # coverage and method mix
dun report        # local HTML report

Collection is local. The hooks, daemon, and ingest command read git plus local agent transcripts and write a local SQLite journal. dun report renders a self-contained HTML file without a server or network call.

The second piece is optional: a DevLake and Grafana layer for teams that want a shared view. whodunit writes to its own whodunit_* tables and leaves DevLake's domain tables alone. If DevLake is already where your delivery metrics live, this lets attribution sit beside them instead of creating another stack.

The supported adapters today are Claude Code, Codex CLI, and agy from Antigravity. Each agent records different things, and whodunit keeps those differences visible instead of smoothing them away.

Privacy, plainly

The collection path makes no network calls.

There are two pieces of identifying information worth naming:

File paths, because they reveal what someone worked on.
Committer email, because a shared dashboard needs to attribute work to contributors.

Both already exist in or around the development workflow. Both stay local unless you configure sync.

The journal has no field for prompt text, message content, file contents, hostnames, or remote URLs. That is not a filtering promise. It is a schema choice. There is nowhere for those values to go.

Where I landed

I built whodunit because I wanted a better answer in the room.

Not a louder answer. Not a prettier one. A more honest one.

Most of the eleven questions can be answered from commits and local transcripts. Adoption spread, model mix, acceptance, autonomy, churn, purpose, and coverage are all measurable with real denominators. Cycle time can be measured when delivery data is connected.

But whether a developer truly understands compaction or model choice is not in the data. And the clean productivity percentage people want is not there unless you are willing to make assumptions quietly.

I am not willing to make them quietly.

So the tool shows the difference, prints the denominator, and leaves room for argument. That is the part I care about most. If the metric logic is wrong, I want to know. If the 1.25x cache write break-even is off, tell me. If there is a better way to handle selection bias, I would genuinely like to hear it.

The point is not to win the AI adoption story.

The point is to make the story sturdy enough to trust.

Code: github.com/navjyotnishant/whodunit

Docs: navjyotnishant.github.io/whodunit

Relayent: use your AI subscription in your app — during dev cycle

Navjyot N — Mon, 20 Jul 2026 02:17:06 +0000

Relayent — a blend of relay and agent. It relays your app's calls to an AI coding agent (Claude Code, Codex, Gemini, Cursor) that's already installed and signed in on a machine you control.

Try it: there's a live playground at relayent-demo.ignorelist.com — pick a model, send a message, and the reply comes back from a real CLI subscription with no API key behind it.

I built Relayent to scratch a very specific itch, and I want to be upfront about its scope before anything else: this is a tool for the development cycle, not for production. When you ship to real users, provisioned API tokens are the right answer — that's what they're for. What follows is about the awkward middle: the weeks of building and QA before you ship, where you're burning metered tokens just to see if the thing works.

The gap: you pay for the model twice

You (or your engineers) already pay for an AI subscription. Claude, Codex, Gemini, Cursor — flat monthly price, and generous. You can lean on it all day in your editor and the bill doesn't move.

Then a feature lands on your plate: a chat box in the product, an in-app assistant, a summarizer that fires on every uploaded doc. And none of it can touch that subscription. The subscription is scoped to the IDE-and-CLI experience — it was never meant to sit behind arbitrary application traffic. So the app talks to the metered API instead, billed per token.

Here's the part that stings. You're not even in production yet. You're developing. Every test run, every "does this prompt actually work," every QA pass over the new feature — all of it draws down metered API tokens, for a feature that isn't earning anything yet. You end up paying twice for the same models: once for a subscription only a human can use, and again, at API rates, to build and test the feature before a single customer sees it.

Relayent's whole premise is that during that phase, the gap doesn't need to exist. There's an authenticated CLI session already sitting on your dev machine. Why can't your app-under-development borrow it? Relayent routes the app's AI calls to that already-running subscription. It doesn't proxy an API key or store a credential — it never touches the CLI's auth at all. It just uses the session that's already there.

To be clear about where the line is: use this to develop and QA against real models without watching a token meter spin. When the feature goes live, wire up proper API tokens. Relayent is a dev-cycle convenience, not a production dependency, and I'd push back on anyone trying to make it one.

How it works

There are just two moving pieces:

The relay — a small job broker that both sides can reach over the network. Your app POSTs jobs to it and long-polls for results. Where it lives is up to you: a public host, an internal box on your intranet, or even localhost on the same machine as the bridge.
The bridge — runs on your own machine, right next to the signed-in CLI. It dials out to the relay, pulls jobs meant for you, runs the AI coding agent headlessly, and posts the result back.

The bridge is where the security story really lives. It only ever makes outbound connections — it opens nothing for the relay (or anyone else) to connect back into, so there's no port to scan, no service to attack on your machine. And crucially, your AI subscription and API keys never leave that machine. The bridge uses the CLI's own already-signed-in session locally; it doesn't read your credentials, doesn't copy them, and doesn't send them to the relay. The relay only ever sees a job going in and a result coming out — never a token, a password, or a subscription. So even a compromised relay never gets your credentials; the worst it can do is queue jobs, not walk away with the keys to your subscription.

So the flow is small on purpose. Your app POSTs a job to the relay; the bridge — on the machine with the logged-in CLI — dials out, polls for work, runs the CLI headlessly, and returns the answer.

  Your app  ──POST──▶  Relay  ◀──poll out──  Bridge
              job    (network-reachable)      (your machine, dials OUT)
                                                     │
                                                     └─▶ claude / codex / gemini / cursor

The shape does two useful things. Nothing listens on the dev machine — no inbound port, no tunnel, no NAT to punch through. The bridge is purely a client; it reaches out, never the reverse. And the relay is the only component anything connects to, so it's the only attack surface to harden — and if you expose it to the public internet, the only thing that has to be internet-hardened. No credentials pass through Relayent itself; the CLI on the bridge uses its own session, and the relay never sees a password or an API key.

The relay is a small Go job broker behind a versioned /v1 API — POST a job, long-poll the result. Four backends work today: claude, codex, gemini, and cursor. (Cursor runs in --mode ask, read-only, so a job can't edit files or run shell commands on the bridge.) Install and sign into whichever CLI you want to lend out, and it shows up — no restart.

The demo playground

The fastest way to get the idea is to go use it: relayent-demo.ignorelist.com.

Note: on the public demo, only Cursor works out of the box — the other backends (Claude, Codex, Gemini) are switched off to keep strangers from spending paid quota. Pick Cursor to see a live reply.

It's a plain chat page — pick a model, type, get a reply. Nothing fancy, and that's the point: every reply on that page is running on a real CLI subscription behind a bridge, not on any API key. The model dropdown isn't hardcoded either; it's populated live from the relay's capabilities, so it only ever lists what a connected bridge can actually run.

Under the hood the demo is a thin proxy. The browser only ever talks to the demo server's own /api/* endpoints, which forward to the relay with a server-side credential — the app credential never reaches the page. And which models the demo is allowed to offer is a policy decision made in the admin console (more on that next). For a public demo like this one, you keep the paid backends switched off and expose something you don't mind strangers exercising — otherwise a public URL is just a way to let the internet spend your quota.

The admin panel

Once more than one person is using a relay, you need somewhere to run it from. That's the admin console.

You sign in with your identity provider — Google, or any OIDC issuer — and land here. It's organized into a few groups:

Users — everyone with a bridge on the relay: their role, whether their bridge is online, how many jobs are pending, how many bridges they've bound. Promote, demote, disable, or remove someone from the same row.
Enrol a bridge — mint a one-time token so a specific user can pair their machine. That's how a bridge gets bound to exactly one person.
Backends — the on/off switches for claude / codex / cursor / gemini. This is the control that keeps a public demo from touching your paid subscriptions: flip the expensive ones off and they simply aren't reachable.
App credentials — issue and revoke the credentials your apps (and the demo) use to enqueue work.
Audit — a per-user activity trail: job counts, backends, timing. Notably, not the prompts or results — the console shows what happened, never what was said.

One deliberate choice worth calling out: the admin is an operator, not an observer. You can see that Alice ran forty jobs on Cursor yesterday and that her bridge is online right now. You cannot see what she asked. That boundary is structural, not a UI toggle — there's simply nowhere in the audit record for prompt content to live.

The console is the admin's view. Everyone else gets their own: sign in as a regular user and you land on a status page scoped to you — is my bridge online, which backends can it run, how many jobs are queued — and nothing about anyone else. Same sign-in door (/login), routed by role: admins to the console, users to their own page. It's the same operator-not-observer line drawn one level down — you can see your own machine, not the relay's other tenants.

The interesting part: keeping tenants apart

Everything above is easy with one user. It gets interesting the moment several people share a relay, each bringing their own subscription — which is exactly the setup where this is useful for a team. Now the relay is holding several authenticated sessions at once, and it has to guarantee that Alice's prompt can never be picked up by Bob's bridge. If it could, Alice would be quietly spending Bob's quota, on Bob's machine, and Bob would never know.

The tempting shortcut is a single shared pairing key — but that quietly makes one secret do three different jobs at once: authenticating the caller, identifying the tenant, and naming the routing namespace. Possession becomes authorization, and anyone holding the key is you: no way to tell tenants apart, no way to revoke just one. That's exactly the property you can't have when several people's subscriptions sit behind one relay.

So Relayent pulls those three jobs apart. Every request carries a Principal:

type Principal struct {
    UserID string
    Kind   string
    Scopes []string
    KeyFP  string
}

Auth decides who you are; the queue routes purely on UserID. And the queue is keyed by that id all the way down, so one person's jobs and another's are never even in the same list:

if q.jobUser[id] != userID {
    return false // unknown job, or another tenant's — same answer either way
}

That check sits at every method that touches a job by id. Knowing a job id isn't enough to do anything with it — you also have to be the user it belongs to, resolved from your Principal, not from something you put in a request body. And it fails identically whether the job doesn't exist or just isn't yours, so probing can't even confirm a job exists in a namespace you can't see.

On security, briefly

I'll keep this short, because it's a blog post and not a threat model. Relayent was built security-first: the bridge only ever dials out (nothing to attack on the user's machine), no credentials pass through the relay, the pairing key is a bearer secret checked in constant time and never logged, humans sign in via OIDC so there's no password anywhere at rest, and machine credentials are stored hashed. Per-user isolation is enforced and tested, anti-spoofing guard included.

The one honest caveat I'd want any reader to take away: isolation here is authorization, not encryption. Tenants can't reach each other's jobs, but the relay operator's process still holds prompts and results in memory in the clear while a job runs. Don't push secrets through a relay you don't trust. If you want the full picture — including what it deliberately doesn't protect against — SECURITY.md lays it out without spin.

Where this lands

Relayent solves one problem well: during development and QA, let your app lean on the AI subscription you're already paying for, instead of quietly metering API tokens on a feature that isn't live yet. Point your team at it, give everyone their own bridge, and the dev-cycle token bill largely goes away.

And it all runs in your own environment. You can stand the whole thing up locally — relay and bridge on machines you control — so nothing leaves your network and no data goes to a third party. You never hand Relayent a key or a subscription; you just install and sign into the AI coding agents you already use (Claude Code, Codex, Gemini, Cursor), and the bridge picks them up automatically. Your credentials stay where they are, your prompts and results stay in your environment, and full data privacy is the default, not a configuration step.

Just don't mistake it for a production answer. In prod, provisioned API tokens are the right tool — properly scoped, properly billed, properly supported. Relayent is the thing that saves you money on the way there.

Try it in your own environment

The demo is the two-minute version. If you want to actually run it — stand up a relay, pair a bridge on your own machine, and point a dev build at it — everything you need is in the repo:

INSTALL.md — start to finish: build the bridge, run a relay (localhost, private, or public with automatic TLS), verify it, and troubleshoot. Exact commands, expected output.
README.md — the overview, quick start, and the multi-tenant setup if you're running it for a team.
API.md — the /v1 contract for wiring your app in: every call, what the numbers mean, and a runnable client.
SECURITY.md — the full threat model and, importantly, what it deliberately doesn't protect against. Read this before you put a relay on the internet.

A local relay + one bridge takes a few minutes and needs nothing but the CLIs you already have signed in. Kick the tires at relayent-demo.ignorelist.com, and the code's on GitHub.