DEV Community: Diya

4 tools I built because I don't trust AI to check AI's work

Diya — Sat, 18 Jul 2026 16:06:28 +0000

If you've spent any real time working with coding agents, you've
probably watched one of them touch a file that had nothing to do with
the task. Or seen a schema quietly change underneath a tool you
depend on. Or tried to figure out, after the fact, which one of 500
subagents actually caused a run to go sideways.

The obvious fix is "have another AI check it." I didn't want that fix.
An LLM judging LLM output costs tokens on every single run, its
reasoning is a black box you can't fully audit, and it can be wrong in
ways that are hard to catch precisely because it sounds confident.

So I built four small tools that do the boring, deterministic version
of the same job instead — AST parsing, graph algorithms, statistics.
No model in the loop. Same answer every time, for free.

TL;DR

Tool	Catches	Cost per run
skillcheck	broken `SKILL.md` files agents rely on	$0
mcp-schema-watch	breaking changes in MCP servers you depend on	$0
pr-blast-radius	PR scope creep, via import-graph analysis	$0
swarm-trace-viewer	where a large agent swarm actually diverged	$0

All four are MIT licensed. Here's what each one actually does.

skillcheck — a linter for the files your agents read

Claude Code, Codex, and Cursor agents all read SKILL.md files to
learn how to do something. Nobody lints those files. skillcheck does:
missing frontmatter, broken relative links, descriptions that are too
short to be useful or too long to fit in context comfortably, and a
hard token budget so one skill file can't quietly eat half an agent's
context window before it's done anything.

Ships as a CLI and a composite GitHub Action, so it runs in CI the
same way ESLint does.

github.com/DIYA73/skillcheck

mcp-schema-watch — breaking-change detection for MCP servers

MCP servers can change their tool schemas without any warning, and if
your code depends on one, that shows up as a silent failure in
production, hours after the actual change happened. This polls the
servers you configure, snapshots each tool's schema, and diffs it
against the last snapshot on every run — classifying every change as
either breaking (a required param appeared, a param was removed,
a type changed) or informational (a description changed, an
optional param was relaxed). Slack only fires on the former.

Postgres for history, BullMQ for scheduling, a small REST API to
manage what's being watched.

github.com/DIYA73/mcp-schema-watch

pr-blast-radius — scope creep, proven with a graph, not a vibe

This is the one I'm most attached to. It parses the real AST of every
file in your repo, builds the import graph, and checks whether the
files changed in a PR are actually connected to each other:

Blast radius (connected-components mode)

  ✓ src/userService.ts
  ✓ src/utils.ts
  ✗ src/paymentService.ts  Not import-connected to the main
                            group of changed files (group size: 1)

3 source file(s) changed, 1 flagged

Two files that import each other are obviously part of the same
change. A third file with no import relationship to either one, sitting
in the same PR? That's worth a second look — and unlike a model's
opinion, this verdict comes from the actual dependency graph, so it's
the same answer every time you run it.

It also excludes high fan-in "hub" files (a shared types.ts, a
barrel index.ts) from the connectivity check first — otherwise every
file in a codebase that imports the same central file looks
"connected," and the whole check becomes noise. Runs as a GitHub
Action that comments on the PR and updates that comment on every
push, instead of leaving a trail of duplicates behind.

github.com/DIYA73/PR-Blast-Radius-

swarm-trace-viewer — finding the one agent that actually broke

The newest of the four, and still just the foundation layer. When an
orchestrator fans out into hundreds of subagents, a run failing
doesn't tell you much on its own — was this agent the actual cause,
or just downstream of something that failed three levels up?

This builds the full agent tree from a flat event list, then walks it
to separate root-cause failures from cascades, and flags
statistical outliers among sibling agents — the one subtask that took
10x longer than its five siblings for no obvious reason. It ships with
a fully deterministic synthetic trace generator, seeded so the same
input always produces the same trace, since there's no public
1,000-agent orchestrator to record real traces from yet.

Live streaming and the actual tree/timeline UI are next.

github.com/DIYA73/swarm-trace-viewer

The pattern

None of these are clever. That's kind of the point — a z-score, an
import graph, and a tree walk will give you the same verdict at 2am as
they will at 2pm, and they'll do it without a bill. As agents write
and orchestrate more of our code, I think there's real room for tools
that check that work the boring way.

All four repos are open for issues and PRs. If one of these solves a
problem you actually have, I'd genuinely like to know.

I built an MCP server that tracks your Claude Code context budget in real time📎

Diya — Sun, 05 Jul 2026 23:51:25 +0000

Every developer using Claude Code or Cursor has hit this wall: the agent starts looping, responses get worse, and then — context overflow. You had no warning. You had no visibility. You just lost the whole session.

I built ContextPulse to fix that.

What is ContextPulse?

ContextPulse is a framework-agnostic MCP server that sits between your AI coding agent and your tools. It intercepts every tool call, counts tokens in real time, tracks your context budget, and fires alerts before you hit the wall.

It ships in two repos:

contextpulse-mcp — the MCP server + NestJS REST API + WebSocket gateway + BullMQ alert queue
contextpulse — the Next.js real-time dashboard

The problem it solves

Current observability tools (Langfuse, LangSmith, Phoenix) are framework-first and show you cost after the run. They don't tell you during the run that you're at 87% context and one tool call away from overflow.

And nobody in open source lets you compare two agent runs side by side to see what changed after a prompt edit.

ContextPulse does both.

Features

Phase 1 — MCP server core

Intercepts tool calls via the MCP protocol
Counts tokens using tiktoken (cl100k_base)
Stores everything in PostgreSQL — sessions, runs, tool calls, budget snapshots, alerts
Fires warnings at 70% and critical alerts at 90% context usage
Detects loops: when the same tool is called 3× in a row

Phase 2 — Real-time dashboard

NestJS REST API: GET /api/runs, /api/runs/:id, /api/runs/:id/tool-calls, /api/runs/:id/alerts
WebSocket gateway (Socket.io) pushes live events to the dashboard
Next.js 15 dashboard with live budget bar, tool call waterfall, alert feed, event log

Phase 3 — BullMQ alert queue + loop graph

BullMQ async alert worker: processes budget and loop alerts without blocking the MCP server
Optional webhook delivery via ALERT_WEBHOOK_URL
Loop detection frequency chart — highlights tools called above the loop threshold in red

Phase 4 — Run diff engine

GET /api/diff?runA=<id>&runB=<id> compares two runs
Returns token delta, tool call delta, budget winner, new loops introduced, loops resolved
Full diff UI in the dashboard: side-by-side stat cards, delta table, tool-by-tool breakdown

Tech stack

TypeScript · NestJS · Next.js 15 · PostgreSQL · Redis
BullMQ · WebSocket (Socket.io) · MCP SDK · tiktoken · Zod · Tailwind

Quick start

1. Start PostgreSQL

createdb contextpulse

2. Clone and run the MCP server

git clone https://github.com/DIYA73/contextpulse-mcp
cd contextpulse-mcp
npm install
cp .env.example .env
npm run dev:api   # NestJS API on port 3000
npm run dev:mcp   # MCP server on stdio

3. Add to Claude Code

// ~/.claude/settings.json
{
  "mcpServers": {
    "contextpulse": {
      "command": "npx",
      "args": ["tsx", "/path/to/contextpulse-mcp/src/proxy/server.ts"],
      "env": {
        "DATABASE_URL": "postgresql://localhost:5432/contextpulse"
      }
    }
  }
}

4. Run the dashboard

git clone https://github.com/DIYA73/contextpulse
cd contextpulse
npm install
echo "NEXT_PUBLIC_API_URL=http://localhost:3000" > .env.local
npm run dev   # Dashboard on port 3001

Open http://localhost:3001 — you'll see the live dashboard.

Using it in your agent

cp_start_session   → get sessionId
cp_start_run       → get runId
cp_track_tool_call → call after every tool call
cp_get_budget      → check live budget anytime
cp_get_run_summary → full run summary with timeline
cp_end_run         → clean up

Example response from cp_track_tool_call:

{
  "toolCallId": "a1b2c3...",
  "inputTokens": 142,
  "outputTokens": 87,
  "totalTokens": 229,
  "budget": {
    "used": 14820,
    "limit": 200000,
    "percentUsed": 7.41
  },
  "budgetStatus": "ok",
  "alert": null
}

When budget hits 90%:

{
  "budgetStatus": "critical",
  "alert": "critical"
}

The run diff engine

This is the feature I haven't seen anywhere else in open source.

After changing your prompt or agent config, you can compare two runs side by side:

GET /api/diff?runA=<uuid>&runB=<uuid>

The response tells you:

Which run used fewer tokens (and by how much %)
Which tools appeared only in one run
New loops introduced in Run B
Loops that were resolved from Run A
Per-tool token delta

The dashboard shows it as a full diff UI with a summary sentence like:

"Run B used 12,400 fewer tokens (23% more efficient). Run B resolved 1 loop: read_file."

What I learned building this

Context engineering is the real bottleneck. Not model capability. The GitHub MCP server alone costs 42,000–55,000 tokens in tool definitions before you type a single prompt. Tracking this in real time changed how I think about agent design.

BullMQ graceful fallback matters. The alert queue degrades to console logging when Redis isn't available. This made the MCP server safe to use without Redis — alerts still fire, they just aren't queued.

TypeScript strict mode with exactOptionalPropertyTypes catches real bugs. The BullMQ + ioredis version conflict (two different Redis class definitions) only showed up clearly because of strict typing.

Repos

MCP server + API: github.com/DIYA73/contextpulse-mcp
Dashboard: github.com/DIYA73/contextpulse

If this is useful to you, a ⭐ on GitHub goes a long way. And if you build something on top of it, I'd love to see it.

I opened my first PR to LiveKit's agents repo — here's the bug I found

Diya — Sun, 21 Jun 2026 18:22:07 +0000

I've been growing my open source portfolio one contribution at a time, and this week I landed on something genuinely interesting in livekit/agents (11k+ stars, the framework behind a ton of real-time voice AI agents).

The bug

If you're building a voice agent on a realtime model (OpenAI Realtime, xAI, Gemini Live), the model streams your transcription back in chunks. A single utterance can fire many user_input_transcribed events before it's final — token by token for OpenAI/xAI, or as one big interim blob for Gemini.

If you want to react exactly once per utterance (say, show a "user is typing" indicator on your frontend via RPC), you need a stable key to correlate all those interim events together.

That key already existed internally — InputTranscriptionCompleted carries an item_id. But when the framework re-emitted it upward as the public UserInputTranscribedEvent, the item_id was silently dropped — leaving consumers with no reliable way to dedupe across providers.

The fix

Small once you see it: add the field, forward it.

class UserInputTranscribedEvent(BaseModel):
    transcript: str
    is_final: bool
    item_id: str | None = None  # new
    ...

def _on_input_audio_transcription_completed(self, ev: llm.InputTranscriptionCompleted) -> None:
    self._session._user_input_transcribed(
        UserInputTranscribedEvent(
            transcript=ev.transcript, is_final=ev.is_final, item_id=ev.item_id
        )
    )

Two files, about 10 lines of real change. The actual work was tracing the event from the realtime model layer, through AgentActivity, up to AgentSession, to find exactly where the field got swallowed.

The takeaway

I didn't need to understand all of livekit-agents to land this — just one event's lifecycle, end to end. Small, well-scoped issues are the most achievable way into a big codebase, especially when someone's already mapped the territory in the issue itself.

PR is up, CI green, waiting on review: github.com/livekit/agents/pull/6172

8 Claude Code skills for NestJS — drop in and use immediately

Diya — Sat, 13 Jun 2026 21:06:26 +0000

Working with NestJS daily, I kept writing the same patterns.

So I packaged them as Claude Code skills — drop into ~/.claude/skills/
and Claude Code picks them up automatically.

What's included

nestjs-module — scaffold any module, service, controller, DTOs correctly in one shot

typeorm-patterns — entities, relations, migrations, QueryBuilder patterns

bullmq-queue — job queues, processors, retry logic, cron jobs

websocket-gateway — Socket.io gateways, rooms, Redis pub/sub streaming

nestjs-auth — JWT auth, guards, role-based access, CurrentUser decorator

redis-caching — cache-aside pattern, TTL strategy, invalidation on write

docker-nestjs — production Dockerfile, docker-compose with health checks

nestjs-testing — unit tests, e2e tests, mocking repos and queues

Install

git clone https://github.com/DIYA73/nestjs-skills ~/.claude/skills/nestjs-skills

Then just describe what you need in Claude Code:

scaffold a users module with TypeORM
add a BullMQ email queue with retry logic  
protect this route with JWT auth
write unit tests for UsersService

GitHub: https://github.com/DIYA73/nestjs-skills

Built these while working on mcp-hub — a self-hosted MCP server
registry also in NestJS. The skills came out of that work.

I built a self-hosted MCP server registry and proxy in TypeScript

Diya — Sat, 13 Jun 2026 16:01:36 +0000

The official MCP registry is written in Go and cloud-hosted.
I wanted something I could self-host, written in TypeScript,
that my team could run inside our own infra.

So I built mcp-hub.

What it does

Register any MCP server (SSE, HTTP, stdio transports)
Proxy all tool calls through one endpoint with auth forwarding
Stream every tool call live via WebSocket (Socket.io)
Monitor server health async via BullMQ jobs

Stack

NestJS — API framework
PostgreSQL — server registry + tool call logs
Redis — WebSocket pub/sub + BullMQ transport
BullMQ — async health check queue
Socket.io — real-time log streaming

How it works

curl -X POST http://localhost:3000/api/v1/servers \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "github-mcp",
    "url": "http://my-mcp-server:8080",
    "transport": "http"
  }'

Then proxy any tool call through mcp-hub:

curl -X POST http://localhost:3000/api/v1/proxy/{serverId}/call \
  -H 'Content-Type: application/json' \
  -d '{ "tool": "create_issue", "input": { "title": "Bug" } }'

Every call is logged to PostgreSQL and streamed live over WebSocket.

Run it in one command

git clone https://github.com/DIYA73/mcp-hub
cd mcp-hub
cp .env.example .env
docker compose up

API at http://localhost:3000/api/v1
WebSocket at ws://localhost:3000/logs

Why I built this

I'm running multiple MCP servers locally and needed a central
place to register them, forward calls, and see what's happening
in real time. Nothing TypeScript-native existed.

GitHub: https://github.com/DIYA73/mcp-hub

Feedback welcome — especially if you're running MCP servers in prod.

I built a self-hosted "Git for AI prompts" — version control, test runner & A/B testing (open source)

Diya — Thu, 11 Jun 2026 23:49:31 +0000

If you're building anything with LLMs, you've probably hit this problem: your prompts are scattered across code files, Notion docs, and Slack threads. There's no version history. You can't easily test a change. And you have no idea what's actually running in production.
I built Prompt Vault to fix this — a self-hosted web app that treats your prompts like code.
What it does
Version control — every edit creates a new version. You can activate any version at any time, just like git checkout.
Auto-detected variables — use {{variable}} syntax in your prompts. The UI automatically detects them and lets you fill values before testing.
Test runner — run any prompt version directly against OpenAI from the UI. See the response, latency, and token usage instantly.
A/B testing — create a test between two prompt versions, run both in parallel, and compare outputs side by side.
Multi-user — JWT auth, each user's data is fully isolated.
Self-hosted — Docker Compose for one-command setup. Your prompts stay on your infrastructure.
Tech stack

Backend: NestJS + TypeScript
Database: PostgreSQL + Prisma
Frontend: Next.js 15 + Tailwind CSS
Auth: JWT + bcrypt
AI: OpenAI API

Quick start
bashgit clone https://github.com/DIYA73/prompt-vault
cd prompt-vault/backend
cp .env.example .env

Add your DATABASE_URL and OPENAI_API_KEY

npm install
npx prisma generate
npx prisma migrate dev --name init
npm run start:dev

New terminal

cd ../frontend
npm install
npm run dev
Open http://localhost:3000 — register, create a project, add your first prompt.
Why I built it
Every team I've seen building LLM features manages prompts the same broken way — hardcoded strings in source code, no history, no way to safely iterate. Prompt Vault is the tool I wished existed.
GitHub
⭐ https://github.com/DIYA73/prompt-vault
Would love feedback — especially from anyone managing prompts at scale.