DEV Community: Ofekv

Your AI agent says "done". Here is what I check before I believe it

Ofekv — Wed, 04 Nov 2026 08:13:48 +0000

⭐ GitHub: github.com/grloper/plumbgraph · open source (Apache-2.0) · by @grloper

grloper / plumbgraph

Local code intelligence and a pre-submit verification gate for AI coding agents: dead code, hallucinated dependencies, weakened tests. CLI + MCP.

Plumbgraph

Know your code. Verify the change.
Local code intelligence and a pre-submit gate for AI coding agents, over CLI and MCP.

Real output of plumb verify on examples/demo-app (captured by scripts/render-demo.py; raw text in docs/demo/verify.txt).

Why

AI agents write code fast and check it badly. Plumbgraph answers the questions an agent should ask before it says "done", and says how sure it is:

Is this code used? dead-code, with reachability evidence and false-positive risks.
Is this dependency real? check-deps compares imports with your manifests and, optionally, PyPI / npm / crates.io, so a hallucinated package is caught.
Did the tests get weaker? weakening reads git diff for deleted tests, new skips and loosened assertions.
What breaks if I change this? impact lists transitive callers and the tests to run; map gives a token-budgeted overview.
One verdict. verify merges all of that with diagnostics, semgrep and ast-grep…

View on GitHub

AI coding agents write code quickly and verify it badly. After a few months of watching them work on my own projects, the same three failures kept turning up, and none of them are caught by "the tests pass". So I built a small open-source tool, Plumbgraph, that checks them on the diff before the agent is allowed to say it is done.

Real output of plumb 1.1.0 (built from the v1.1.0 tag) on a throwaway copy of the Python CLI of my other project, PlayGoblin: 35 Python files. The clean copy passes. Then I made an edit by hand that imitates an agent (a fake import, an unused helper, a skip marker, a deleted assertion) and plumb verify fails. The edit is simulated by me; the output is not. The GIF is a replay of captured text, and long lines are cut in the picture; the verbatim output is in docs/media/demo-output.txt.

The three failures

Packages that do not exist. The agent imports something plausible and the install fails, or worse, someone registers the name later. A USENIX Security 2025 study cited in the repo's docs/STRATEGY.md reported that 19.7% of package recommendations from 16 models were hallucinated; I have not reproduced that number, I just find it reason enough to check.

Tests that got weaker. Deleted tests, new skips, loosened assertions. The suite is green because the suite shrank. A diff-level check catches this subset of test tampering (not every kind, for example special-casing inputs).

Code nothing calls. Agents add helpers and forget them.

What Plumbgraph does

plumb indexes your repo with tree-sitter into a local SQLite file (Python, JS/TS, Rust, Go, Java, C#). It executes none of your code by default. Every result carries a source and a confidence, so an agent (or you) can tell a guess from compiler-grade evidence.

check-deps: imports vs your manifests, and with --online, against PyPI, npm and crates.io. Registry errors are never reported as "does not exist".
weakening: reads git diff for deleted tests, new skips and loosened assertions. On seeded commits it flagged 12/12 on Python, JS, Rust, Go and C# with 0 of 3 benign changes flagged per repo. I seeded those myself, so read that as a regression check, not a benchmark.
dead-code: reachability from entry points and tests, with evidence and false-positive risks per finding.
map and impact: a PageRank-ranked overview under a hard token budget, and transitive callers plus the tests to run.
verify: merges all of it (plus diagnostics, semgrep and ast-grep with --run), diffs against a baseline so only new findings count, and returns one verdict.

The four findings above are what the tool printed for my simulated edit, each with a why: and a risk: line. The risk: lines matter: the skip marker could be a legitimate platform skip, and a Python import name can differ from its package name. The tool says so instead of pretending to be sure.

map and impact are the other half, the part an agent uses before it edits. On the same copy, plumb impact load_run lists 68 affected symbols across 11 files and the six test files to run, each edge with a confidence that drops with distance:

A gate that fails closed

The verdict is the part I spent the most time on. pass means the selected checks completed and nothing new met its threshold. fail means new enforcing findings. incomplete means a required check failed, timed out or could not run, or the Git base is missing; it never passes. Dead code is advisory by default because an index cannot prove an external caller, reflection or safe deletion. It is a gate over the checks you selected, not proof the change is correct. In the clean run of the demo, two optional checks were skipped, and the output says so rather than counting them as passed.

For MCP, an isError: false response is not a pass: you must check data.verdict == "pass". And the agent cannot grant itself powers: running external tools needs --allow-exec and writing baselines needs --allow-baseline-update, both set by the operator when the server starts. Results are marked untrusted: true, because text from repository files is data, never instructions.

Quickstart

git clone https://github.com/grloper/plumbgraph && cd plumbgraph
git checkout v1.1.0
cargo install --path crates/cli --locked      # Rust 1.85+; installs `plumb`

plumb map . --tokens 1500
plumb check-deps . --online                   # sends package names only
plumb verify .                                # needs a Git base
claude mcp add plumbgraph -- plumb mcp --root /path/to/project

plumb init --dry-run previews the AGENTS.md block and MCP config it would write. The AGENTS.md block is deliberately minimal: a study cited in the docs found verbose context files can reduce agent success and raise cost.

For this article I built it again on a Windows machine with the GNU toolchain. It needed a MinGW gcc on PATH (the ring crate failed to compile without it), and it then built in about a minute.

What real repos taught me

I ran it on requests, express, ripgrep, cobra, gson, Newtonsoft.Json, kubernetes and VS Code. The first version took 640 s to index kubernetes and ran out of memory on later commands; after fixes (PR #14: edge dedup, an ambiguity cap, a resolver fix), the index takes 41 s and commands finish in 6-13 s on a 16 GB Linux box. Running it on one of my own Unity projects exposed false positives (Unity message methods on partial classes), which are fixed in v1.0.3. Using v1.0.0 on Windows showed that every provider was reported missing because lookup ignored .exe and .cmd. None of that shows up on toy fixtures.

Left: docs/bench/results-2026-10-10.json, measured by scripts/bench.py with an older build (plumb 0.2.0) on an 8-CPU Linux machine, one run per repo. Right: docs/VALIDATION.md. These are not comparisons with other tools, and I have not measured any other tool.

Honest limits

Without a SCIP index the analysis is name-based: no types, and common names give lower-confidence edges.
Dead-code precision on libraries in the default app mode is low (use --lib). A hand review of 76 findings on chi and gson found 1 real unused function. Recall is unmeasured on real code.
Reflection, eval, DI containers and macros are invisible.
MCP: stdio only. I tested with the official TypeScript SDK, the inspector CLI and one agent host on Windows. I have not tested Claude Code, Cursor or Codex end to end.
Windows was exercised by hand on one machine and CI is Linux only. There are no prebuilt binaries and no crates.io package.
I have not run an experiment showing that the tool improves agent outcomes on real tasks. I make no claim about that.
The demo edit above is a hand-made imitation of an agent, on a small repo. It shows what the checks print, not how often real agents make these mistakes.

Try it, and help

Run plumb verify . on a branch an agent wrote and tell me what it got wrong: false positives, false negatives, and anything that breaks in Claude Code, Cursor or Codex. Issue templates for bugs and false positives are in the repo: github.com/grloper/plumbgraph. Packaging (prebuilt binaries, crates.io) and more SCIP indexers are the obvious contributions.

I built a bot that plays my Unreal game and files the bugs

Ofekv — Wed, 04 Nov 2026 06:50:42 +0000

⭐ GitHub: github.com/grloper/playgoblin · open source (Apache-2.0) · by @grloper

grloper / playgoblin

A goblin plays your game overnight and files the bugs it breaks. Open-source QA bot for Unity, Unreal and any engine, with an MCP server for AI agents.

PlayGoblin

An open-source QA bot that plays your game like a person and files the bugs with evidence.
Unity, Unreal, or any engine that can write a log file. Deterministic checks, no LLM in the loop, no account.

A real 150 s run on a UE 5.8 game (3x time-lapse of seconds 19-52, overlay added). Full run: 251 m walked, 19 waypoints, 12.0% coverage, 1 confirmed and 12 unconfirmed stuck spots.

A frame PlayGoblin captured while walking the town level of a real Unreal Engine 5.8 game (Use Your Friends). Frames and clips like this are attached to findings.

Status: pre-release (0.1.0-dev). Works on Unity 6000.4, and on Unreal 5.8.3 (Win64) with a sample project and one real game Expect rough edges. See Status for exactly what has and has not been verified.

What it catches

These come from real runs, not mock-ups. Most were found by running on a…

View on GitHub

I'm making an indie game in Unreal Engine 5.8, and like most solo developers I have a testing problem: nobody walks the whole game after every change. I do it for ten minutes, see nothing wrong, and ship the build. So I wrote a bot to do the boring part, and I'm releasing it as open source. It is called PlayGoblin, and it is pre-release (0.1.0-dev), so please read the limits section before you get excited.

A real 150 s run on my game, shown as a 3x time-lapse of seconds 19 to 52 (overlay added by me). The full run walked 251 m and reached 19 waypoints, but covered only 12.0% of the 1137 walkable cells; it logged 1 confirmed and 12 unconfirmed stuck spots. Coverage in the overlay is derived from sampled positions. The goblin is not a speedrunner yet.

The problem

Unit tests check what you thought to assert. Recorded input replays only cover the path you recorded and break when the level changes. Manual QA is great, but it does not run at 3 a.m. after every commit. The gap I care about is "nobody walked there": a stuck spot behind a prop, a fall through a missing collision, a menu button wired to nothing, a TODO that made it into a title screen.

What PlayGoblin is

An engine-side component walks the game and writes a plain run folder: events.jsonl plus a summary. A Python CLI reads that folder and produces findings, an HTML report, and optionally GitHub issues. Each finding has evidence, meaning an event, a frame, or a log line. If the bot cannot show it, it does not report it.

There are engine sides for Unity (6000.4 tested) and Unreal (5.8.3 Win64 tested). Any other engine that can write the run-folder format (documented in docs/PROTOCOL.md) gets the same checks and reports. I tested that with a fake Godot-style run, not a real Godot game.

The bugs it found on my game

I ran it on my Unreal game this week. Three results are worth telling honestly.

The player could not move. The game had no input bound to move the pawn. The bot, which reads the game's own key and axis bindings and presses the keys that should move the player, never left the spawn point and reported "The player never moved" as critical.

A bug in the bot, found by the game. My game has a level-sized invisible blocking volume. It hid the ground from every floor probe, so the bot saw nothing walkable. That was a bug in PlayGoblin, not in the game, and the fix is that the bot now skips actors it starts inside and logs them. Real games are weirder than sample projects, and this is why I am asking for run folders from yours.

Grey-box look. It lists visible actors still using /Engine/BasicShapes meshes or the engine grid material and reports each as "Prototype look". It is a cheap way to see how much of a level is still placeholder art.

On the Unreal sample project, which has bugs I planted, it also finds a UMG "Options" button that does nothing and menu text reading "TODO: title". Another false positive taught me something: it flagged a pastel-pink game as missing-material pink. Now only pure magenta counts.

How it works

It plays through real input. In key mode (-PlayGoblinKeys) it presses the game's bound keys, turns with simulated mouse movement rather than setting the camera, taps action keys at irregular intervals, and tries a jump when stuck. If nothing moves the player, or the mouse does nothing after repeated probes, that is itself a finding (dead_input).

It crawls UMG menus. About every two seconds it finds visible widgets, reports placeholder text, and clicks each enabled button once. A button is dead if nothing observable changed within a second.

It explores on purpose. Instead of random destinations it heads for the nearest unvisited ground next to what it has already seen. On the sample project, same seed and 150 seconds, coverage went from 34% to 73% on one level and from 48% to 93% on the other. Every report prints coverage.

The checks have no LLM in them. Each check is a pure function: run folder in, findings out. The same events give the same findings every time, and there are 62 Python tests. A stuck spot only counts as confirmed once it repeats, so one unlucky run does not file a fake bug. An LLM can optionally write a summary paragraph, labelled as such.

Agents can drive it. There is an MCP server (playgoblin mcp) with five tools: status, run, report, findings and file_issues. The repo is also a Claude Code plugin with a skill that teaches an agent when to run it and how to read confidence. Filing GitHub issues is a dry run unless you explicitly confirm.

The HTML report: each finding links to its evidence.

Quickstart

pip install "git+https://github.com/grloper/playgoblin#subdirectory=cli"   # Python 3.11+

Unreal: copy unreal/PlayGoblin into your project's Plugins/ and launch the game with the flag:

UnrealEditor-Cmd.exe MyGame.uproject -game -playgoblin -PlayGoblinSeconds=300 -PlayGoblinSeed=1 -PlayGoblinOut=D:/runs -PlayGoblinExit
playgoblin report D:/runs/<run id>     # report.html, report.md, findings.json

Unity: add the package from https://github.com/grloper/playgoblin.git?path=/unity, put the PlayGoblin Player component on your player, press Play, then playgoblin run --minutes 10 --runs 3.

The plugin does nothing unless -playgoblin is on the command line, so shipping with it enabled is harmless.

Honest limits

It is pre-release. Unity 6000.4 and Unreal 5.8.3 on Win64 are the only tested combinations.
Accuracy on real games is not measured. My samples were written by the people who wrote the checks, so treat them as regression tests. The Unreal plugin passed its sample (all 5 planted bugs, nothing on the clean twin) on one seed and one machine.
UMG buttons are clicked by broadcasting OnClicked, so Slate hit testing is not exercised; Enhanced-Input-driven and CommonUI-only menus are not covered.
It walks and clicks. It does not judge fun, balance, visuals or audio, and "no findings" never means bug free.

Try it, and help

The most useful contribution right now is running it on a game and telling me what it missed or got wrong, with the run folder attached. Open an issue at https://github.com/grloper/playgoblin. Roadmap items: more real-game runs with published precision numbers, other Unreal versions and Enhanced Input, Unity UI Toolkit and player builds, and a GitHub Action. If you build for another engine, the run-folder format is short; a writer for it is a good first contribution.

Supercharging DeepSeek Harness: Bringing Claude Pro/Max with One-Click Google/Gmail OAuth Login and Real-Time Quota Tracking

Ofekv — Sun, 06 Sep 2026 11:01:15 +0000

By **grloper* (GitHub @grloper)*

1. Introduction: The Subscription Dilemma in Modern Agent Frameworks

DeepSeek Harness (DSH) has quickly established itself as one of the most capable, modular agent orchestration environments. Built on an extensible Cordis architecture where every feature â€” from persistence and sandboxing to tools and UI components â€” is a hot-pluggable plugin, DSH allows developers to assemble tailored autonomous coding environments.

However, many AI developers face a common friction point: how do you use your existing Claude subscription in DSH?

Millions of developers subscribe to Claude Pro or Claude Max directly through claude.ai. These subscriptions include substantial usage allowances, but they authenticate via browser OAuth rather than raw developer console API keys. Previously, attempting to use Claude in DSH meant:

Being forced to open a terminal, run claude login, and hope local files were parsed correctly.
Manually dumping tokens and editing YAML configuration files.
Dealing with silent token expiry without any visual feedback.
If you originally signed up with "Continue with Google" (Gmail), there was no native UI in DSH Settings to authenticate.

To eliminate this friction once and for all, I built and open-sourced dsh-claude-oauth â€” a complete, production-grade DSH plugin that provides a native Claude Settings section with one-click Google / Gmail sign-in, auto-refreshing tokens, live model discovery, and real-time subscription quota tracking.

2. Visual Walkthrough: How It Looks in DSH

Here is the native Settings interface that dsh-claude-oauth contributes to DeepSeek Harness:

What You See in Settings

Dedicated Navigation Entry: Positioned at order 13 in the Settings sidebar, sitting cleanly alongside Antigravity (Google Gemini) and OpenAI Codex.
Account Status Card: When authenticated, displays your linked Google/Gmail address, subscription tier pill (CLAUDE MAX or CLAUDE PRO), and active connection status.
Live Subscription Quota Bars: Shows real-time utilization for both the 5-Hour and 7-Day subscription windows with live countdown timers until reset.
Active Model Catalog: Displays available models (Claude Sonnet 4.5, Claude Opus 4.5, Claude Haiku 4.5, and Claude 3.7 Sonnet) verified live from Anthropic's API.

3. Architecture Deep-Dive: How It Works

Under the hood, dsh-claude-oauth is divided into two cooperative halves: a Host plugin running in Node.js and a Web Client plugin running in the browser.

3.1 The PKCE Loopback Flow & The Windows IPv6 Pitfall

When you click "Sign in with Google / Gmail":

The client opens a tab pointing to /api/anthropic-oauth/login.
The host generates a cryptographically random 32-byte PKCE code verifier and computes its SHA-256 code challenge (S256), along with a random CSRF state token.
The host binds an ephemeral HTTP callback listener on localhost:0.

Here is a critical engineering detail: Windows localhost resolution.
On modern Windows systems, localhost often resolves to IPv6 ::1 before IPv4 127.0.0.1. A server that only binds to 127.0.0.1 will result in ECONNREFUSED when the browser redirects back to http://localhost:<port>/callback.

dsh-claude-oauth implements a dual-stack listener that binds both IPv4 and IPv6 on the identical ephemeral port:

server4 = http.createServer(handler)
await new Promise((res) => server4.listen(0, '127.0.0.1', res))
resolvedPort = server4.address().port

try {
  server6 = http.createServer(handler)
  await new Promise((res) => server6.listen(resolvedPort, '::1', res))
} catch {}

The user is redirected to https://claude.ai/oauth/authorize.
On the Claude sign-in screen, the user chooses "Continue with Google" and selects their Gmail account.
Once authorized, Anthropic redirects back to http://localhost:<port>/callback?code=...&state=....
The loopback server verifies the state, exchanges the authorization code for tokens via https://claude.ai/v1/oauth/token, and queries https://api.anthropic.com/api/oauth/profile to retrieve the user's Gmail address and subscription tier.

3.2 Multi-Store Credential Synchronization

To prevent isolated silos, dsh-claude-oauth synchronizes credentials across all relevant stores upon authentication:

~/.claude/.credentials.json: Writes the claudeAiOauth session (access token, refresh token, expiry, scopes, email). This ensures the official Claude Code CLI also recognizes the login!
~/.dsh/.credentials.yaml: Mounts ANTHROPIC_OAUTH_TOKEN and CLAUDE_CODE_OAUTH_TOKEN into the native DSH credential resolution seam.
~/.dsh/settings.yaml: Automatically declares llm-pi-ai.providers.anthropic with the live discovered model list.
~/.dsh/plugins/subscriptions/auth.json: Syncs with the multi-provider subscriptions store so other plugins stay updated.

3.3 Real-Time Unified Rate Limit Probing

Anthropic attaches its unified subscription rate-limit headers only to inference responses (POST /v1/messages), never to catalog endpoints (/v1/models).

To provide live 5-hour and 7-day usage readouts without burning quota, dsh-claude-oauth executes an ultra-cheap max_tokens: 1 probe using Claude 3.5 Haiku:

const res = await fetch('https://api.anthropic.com/v1/messages', {
  method: 'POST',
  headers: await claudeCodeHeaders(accessToken, OAUTH_DISCOVERY_BETAS),
  body: JSON.stringify({
    model: 'claude-haiku-4-5-20251001',
    max_tokens: 1,
    messages: [{ role: 'user', content: 'hi' }]
  })
})

From this single-token response, it extracts:

anthropic-ratelimit-unified-5h-utilization
anthropic-ratelimit-unified-5h-reset
anthropic-ratelimit-unified-7d-utilization
anthropic-ratelimit-unified-7d-reset

These values are cached with a 60-second TTL and rendered into smooth, color-coded progress bars (green below 70%, amber 70â€“89%, red 90%+).

4. How to Install and Use

Step 1: Install the Plugin

Run this command from your terminal:

dsh plugin --profile web add github:grloper/dsh-claude-oauth

Step 2: Open Settings & Sign In

Start DSH Web: dsh web
Click the gear icon in the lower-left corner to open Settings.
Click on Claude (Anthropic).
Click "Sign in with Google / Gmail".
Select your Google/Gmail account and click Authorize.
The browser confirms success; return to DSH and see your account and quota live!

Step 3: Start Coding with Claude

Open any session and select Claude Sonnet 4.5 or Claude 3.7 Sonnet from the composer model picker.

To make Claude your default agent model across all sessions, set it in ~/.dsh/settings.yaml:

agent-default-model:
  provider: anthropic
  model: claude-sonnet-4-5

5. Conclusion & Open Source Repository

DeepSeek Harness provides an outstanding foundation for autonomous AI coding. With dsh-claude-oauth, utilizing your existing Claude Pro or Max subscription with Google/Gmail login is now as frictionless as a single click.

GitHub Repository: https://github.com/grloper/dsh-claude-oauth
License: MIT
Contributions: Pull requests, feature suggestions, and bug reports are warmly welcomed!