DEV Community

NIRMEET TRIVEDI
NIRMEET TRIVEDI

Posted on

I audited my own product and found it never actually worked for its own core use case

I built Meanwhile, a status line for Claude Code, Copilot CLI, and VS Code. Most of the time it shows a quiet tip. Sometimes it shows a clearly-labeled "(sponsored)" line instead, and when that happens half of what the sponsor paid goes back to the developer.

This week I actually sat down and audited my own code end to end instead of just checking install counts. What I found wasn't great: the VS Code extension had been silently broken for the exact use case it exists for, since the day it shipped.

Bug 1: the extension required a keystroke every 30 seconds

const ACTIVITY_FRESHNESS_MS = 30_000;

const poll = async () => {
  const windowFocused = vscode.window.state.focused;
  const recentlyActive = Date.now() - lastActivityAt < ACTIVITY_FRESHNESS_MS;
  if (!windowFocused || !recentlyActive) return;
  // ...fetch and bill a line...
};
Enter fullscreen mode Exit fullscreen mode

lastActivityAt only updated on a real document edit. The moment you fire off an agent and take your hands off the keyboard to let it think, recentlyActive goes false within 30 seconds and the extension goes completely dark. That's the exact window the whole product exists to monetize.

Fix: track presence more broadly (tab switches, cursor movement, regaining window focus, not just edits), and widen the window to 5 minutes to actually match how long an agent turn runs.

Bug 2: even a live poll got flagged as fake

Independently, the server had its own gate:

function sessionProgressed(baseline, current) {
  if (!baseline) return true;
  if (current.tokens !== null && baseline.tokens !== null && current.tokens > baseline.tokens) return true;
  // ...
  return false;
}
Enter fullscreen mode Exit fullscreen mode

For VS Code, tokens is mapped to the editor's own edit count. Requiring it to strictly increase makes sense for Claude Code (a real session's cost/tokens always climb turn over turn) but is exactly backwards for VS Code: edit count is supposed to plateau while you're waiting on an agent. So even after fixing bug 1, a genuinely live poll during a wait still got silently discarded as "looks like a faked ping."

Two independent bugs, stacked, both silently killing the same use case. Fixed by trusting a live poll within the same session (matched by session ID) instead of requiring a number that has no reason to move.

Bug 3: Copilot CLI only ever billed the first turn of a session

Copilot CLI's hook payload never sends cost or token data at all, just a session_id — and unlike Claude Code, that id doesn't change per turn, it's scoped to the whole CLI session. The server's fallback logic assumed a changing session id was the signal of real progression. Since Copilot's never changes within a session, every turn after the first billed $0, silently, for as long as the extension has existed.

The smaller stuff

Also found and fixed while in there:

  • /network-stats was doing a full sequential KV scan (600+ reads) on every request — replaced with a self-seeding Durable Object counter, now O(1)
  • A daily payout cron was writing install state directly to KV, bypassing the Durable Object that's supposed to be authoritative — a real double-payout risk
  • Abandoned advertiser checkouts were staying in a shared index forever, costing a KV read on every single status-line poll from every user
  • The Windows installer was missing a required config field the macOS/Linux one had
  • All three client scripts would hang forever if you ran them by hand to test (blocking on stdin with no TTY check)
  • The direct extension download on the site was still serving a build with bug 1 in it

None of this is retroactive marketing spin — if you installed the VS Code extension before this week and it felt dead, that was real, not you. It's fixed now (0.1.7, live on Open VSX).

I don't have big usage numbers to show off here. What I have is a real list of specific, embarrassing bugs, found by actually reading the code instead of trusting a dashboard, and fixed. If that's useful to anyone auditing their own "is this actually doing what I think it's doing" assumptions, that's the whole point of writing this up.

Top comments (0)