DEV Community: Frank Guan

Stop "Vibe Checking" Your AI Agents: How to Build Production Evals in 60 Minutes

Frank Guan — Wed, 30 Sep 2026 16:45:07 +0000

How do you currently test your AI agents?

If you're like most developers building agentic workflows, you probably rely on the "vibe check": tweak a prompt, run a couple of manual queries, check if the output looks reasonable, and assume it's good to go.

Here's why that breaks down fast:

Silent Regressions: A teammate submits a PR or tweaks a tool definition, and you have no way to know if performance improved or degraded.
Cost & Latency Spikes: An agent gets trapped in a multi-turn retry loop, quietly burning thousands of tokens.
The "Looks Right" Trap: Output can look perfectly well-written while omitting critical parameters or hallucinating key instructions.

In this episode of the AI Agent Clinic, Google Cloud engineer Dani Zamora and Matthew Feroz (Merge) tackle this challenge live: building an end-to-end, framework-agnostic evaluation pipeline in 60 minutes.

Check out the full walkthrough below:

What We Cover in the Video

The Test Subject (DocsHound): Matt brings on DocsHound, an open-source LangGraph agent that scans GitHub issues and PRs to find documentation gaps and open automated PRs.
Framework-Agnostic Telemetry (OpenTelemetry + OpenInference): How to standardize multi-turn traces across any agent framework (LangGraph, CrewAI, AutoGen, ADK) with zero vendor lock-in.
The 3-Tier Metric Strategy: Why you shouldn't start with 50 metrics, and how to combine:
- Managed core metrics: Trajectory quality, tool-calling correctness, and groundedness.
- Custom LLM judges: Rubrics tailored to formatting, code block enforcement, and actionability.
- Deterministic assertions & operational tracking: Syntax checks, latency, token usage, and API cost.
The 0.33 Quality Blind Spot: How the automated scorecard caught a 33% documentation quality failure that manual spot-checks completely missed.

Jump Directly to Key Sections

If you want to skip straight to a specific part of the walkthrough:

00:00 — Why "vibe checking" fails in production
02:11 — Introducing DocsHound (LangGraph agent demo)
03:40 — Standardizing traces with OpenTelemetry & OpenInference
05:51 — The 60-minute eval challenge begins
06:47 — Step 1: Mapping agent execution flow with Antigravity
11:18 — Step 2: Scaffolding the open-source Agent-Eval toolkit
17:14 — Balancing quality vs. latency vs. token cost
18:34 — Step 3: Translating quality definitions into custom metrics
21:40 — Step 4: Inspecting the scorecard & spotting the 0.33 failure
24:08 — Why automated evals change how you build agents

Open-Source Repositories & Resources

Follow along with the exact tools used in the video:

📺 Full Video: Watch on YouTube (25 mins)
🛠️ Agent-Eval Toolkit: github.com/google-cloud-tech/agent-eval
🐶 DocsHound Agent Code: github.com/google-cloud-tech/docshound
📖 Enterprise Agent Platform Docs: g.dev/cloud/agent-evaluation

👉 Watch the full video above to see how to set up the eval pipeline step-by-step. If you have questions about instrumenting your own agent framework or setting up custom metrics, drop a comment below!

How we slashed an AI Agent's latency by 80% in 60 minutes

Frank Guan — Wed, 01 Jul 2026 18:44:11 +0000

Building an AI agent is fun. Fixing its production latency when it's juggling live data, RAG, and text-to-speech? Not so fun.

In the latest episode of the AI Agent Clinic, we sat down with developer Sami Maghnaoui to debug PlaybackIQ, a football / soccer agent he built to provide pre and post match analysis with text to voice, and minute-by-minute match insights with interactive UI.

The app was awesome, but under heavy "match day" data loads, the wait times were killing the UX.

Here’s how we fixed it:

The Bottleneck: We implemented OpenTelemetry on the Agent Platform to trace exactly where the LLM calls and data retrieval were hanging up.
The Scale: We shifted the deployment to Cloud Run to properly handle concurrent traffic.
The Result: We managed to slash the agent's latency by 80%.

If you're dealing with sluggish LLM response times in your own apps and want to see what a production-grade fix looks like, we recorded the whole teardown and rebuild.

🎥 Watch the teardown here:

(Let me know in the comments what your go-to stack is for tracing LLM latency!)

Check out this great tutorial on building ADK agent frontends with AG-UI Protocol and CopilotKit

Frank Guan — Mon, 29 Sep 2025 21:56:38 +0000

Ready to build a user-friendly frontend for your ADK agents? This guide will show you how to use CopilotKit and the AG-UI protocol to create a powerful, real-time user interface for your AI agents. 🤖

Bonnie for CopilotKit

Sep 25 '25

Build a Frontend for your ADK Agents with AG-UI

#webdev #programming #ai #tutorial

243

12 min read

Why Use CopilotKit?
Speed up development: Go from idea to a working prototype in just a few hours.

Pre-built UI components: Save time with ready-to-use components like CopilotSidebar.
Real-time updates: Stream your agent's responses and tool outputs for a dynamic user experience.
Simplified state management: The useCoAgent hook makes it easy to sync your agent's state with the frontend.

Getting Started
Here’s a quick overview of the key steps:

Setup: Use a single CLI command to set up a full-stack ADK agent with both the backend (AG-UI) and frontend (CopilotKit) configured.
Backend Integration: Integrate your ADK agent with the AG-UI protocol and expose it as an ASGI application through a FastAPI endpoint.
Frontend Development: Install the CopilotKit packages and configure a Copilot Runtime instance to communicate with the backend.
UI Components: Use the CopilotKit provider to manage agent sessions and CopilotSidebar to create the chat interface.
State Synchronization: Use the useCoAgent hook to share and sync the agent's state with the frontend in real time.
Real-time Updates: Stream and render your agent's responses and tool outputs on the frontend.

With these tools, you can quickly build a production-ready AI agent with a rich and interactive user interface. Worth checking out if you're building AI agents and want rich frontend experiences! Learn more in this post or other resources below:

Quick Start Repo: npx copilotkit@latest create -f adk
Official Documentation: https://docs.copilotkit.ai/adk
Starter Project on GitHub: https://github.com/copilotkit/with-adk
AG-UI Dojo Tutorial: https://dojo.ag-ui.com/adk-middleware/feature/shared_state

Google AI Agent Bake-Off: Episode 1

Frank Guan — Fri, 25 Jul 2025 14:46:57 +0000

Hey DEV community, ever wondered what developers could build with powerful AI tools under pressure? We challenged four teams to find out in the AI Agent Bake-Off series!

Watch as developers, paired with Googler "sous-chefs," vibe with code and build sophisticated AI agents in a 3-hour sprint against the clock. See the innovation in action in the first episode, and tell us: What would YOU build with the Agent Development Kit and Gemini?