DEV Community: Yu-Wei Simon Liu (Simon Liu)

[Open Source] ADEval — A Tool for Evaluating Tool-Use Capabilities of Google ADK AI Agents

Yu-Wei Simon Liu (Simon Liu) — Tue, 10 Mar 2026 04:58:31 +0000

I. Introduction: Why Building a "Stable" AI Agent is Hard

As developers, we all know that scaffolding an AI Agent using Google’s Agent Development Kit (ADK) or various LLM frameworks is relatively straightforward. The real challenge, however, lies in ensuring the Agent's behavior is predictable and stable.

In real-world business scenarios, an Agent might call the right tool today but deviate tomorrow due to slight prompt variations, model updates, or context interference. Relying solely on manual chat testing is inefficient and fails to cover critical edge cases.

This is why I built ADEval — a systematic evaluation tool designed specifically for AI Agents. It empowers developers to gain deep control over Agent behavior through a dual-track approach: Automation and Visualization.

II. Github: ADEval

ADEval provides an intuitive Web UI and a powerful CLI, allowing you to systematically test your Agent's Question-Tools-Answer (Q-Tools-A) flow. It supports experiment management, batch testing, and comprehensive tracing.

Github Repository GitHub - ap-mic-inc/ADEval: Google ADK Evaluation Service
Project Documentation ADEval Documentation

III. Core Philosophy: The Q-Tools-A Validation Framework

When evaluating an AI Agent, comparing the final text response (Answer) is simply not enough. A high-quality Agent must call the right Tools at the right time with the correct parameters.

ADEval is designed around the Q-Tools-A logic:

Question: The input prompt, specific User ID, and necessary Session State (persistence).
Tools: Automatically validates if the Agent invoked the expected tools. We support Smart Argument Comparison — even if the JSON parameter order differs, ADEval marks it as a match if the values and logic are consistent.
Answer: Ensures the final response meets business requirements via keyword matching or semantic checks.

Below is the workflow logic diagram I designed for ADEval, ensuring a clear path from start to output:

ADEval Workflow Logic Diagram (Mermaid Chart)

IV. Dual Mode: From Debugging to Production Automation

ADEval provides both an intuitive Web UI and a powerful CLI, allowing you to switch seamlessly between manual debugging and CI/CD automation.

1. Web UI: Visual Tracing & Real-time Debugging

Observing an Agent's thought process is critical. The ADEval Web dashboard features a powerful "Playground" where you can input questions and observe results in real-time.

ADEval Playground Interface showing testing panels

The standout feature is Visual Tracing. We transform complex API response event streams into a "Dark Terminal Style" viewer. You can expand raw JSON with one click to pinpoint exactly where a tool call failed or deviated.

Visual Tracing Dark Terminal View showing JSON events

2. CLI Tool: Powerhouse for CI/CD & Batch Execution

Once your experiments are defined, you no longer need a browser. ADEval offers a full-fledged command-line tool perfect for automation scripts:

Global Configuration (adeval config): Set default API URLs and developer credentials to save time.
Quick Test (adeval test): Perform stress tests or logic validation directly against the Agent without creating an experiment.
Batch Execution & Reporting (adeval run / export): Execute entire experiment sets and receive precise statistical reports.

CLI output showing 'adeval run' statistical table

V. Deep Dive into Features

Experiment Management & Batch Evaluation: Import dozens or hundreds of test cases via CSV files. ADEval provides real-time progress bars and pass-rate statistics during execution.
Local Data Ownership: Privacy and performance are priorities. All experiment data, logs, and configs are stored locally in the .adeval/ folder. Your data stays on your machine.
Smart Comparison Logic: In tool validation, we support "Order-Independent" comparison. If an Agent calls get_weather(city="Taipei", unit="c"), ADEval accurately judges it even if the internal parameter sequence differs.

VI. Getting Started

ADEval is open-source and ready to use. You can install it easily via Python:

# Clone the repository
git clone https://github.com/ap-mic-inc/ADEval.git
cd ADEval

# Install in editable mode
pip install -e .

After installation, simply run adeval ui to launch the web interface or check out adeval --help for the full CLI suite.

VII. Conclusion

Building an AI Agent that can chat naturally is just the beginning. Ensuring its stability and predictability in complex business scenarios is where the real engineering challenge lies. ADEval was born to solve the core pain points of testing, debugging, and maintaining Agent systems.

In summary, ADEval brings three key values to developers:

Precise Behavioral Control: Move beyond text matching to rigorous Tool-use validation.
Flexible Workflow: Covers the entire lifecycle from visual debugging to automated regression testing.
Security & Efficiency: Localized storage for privacy and smart matching to reduce false positives.

If you are building high-quality enterprise AI applications with Google ADK, ADEval is your indispensable testing companion. Give us a Star (⭐) on GitHub or submit an Issue/PR to help us build a stronger AI evaluation ecosystem!

I am Simon
Hi everyone, I am Simon Liu, an AI Solutions Expert and Google Developer Expert (GDE) in GenAI. I am dedicated to helping enterprises implement AI technologies to solve real-world problems. If this post was helpful, please follow me on Medium or connect with me on LinkedIn.

My Personal Website: https://simonliuyuwei.my.canva.site/link-in-bio

[ Open Source Project] open-translate — Offline Translation Web Service Powered by TranslateGemma

Yu-Wei Simon Liu (Simon Liu) — Sun, 18 Jan 2026 16:26:31 +0000

I’ve provided a Google Colab test script for this project. You can apply for Hugging Face and ngrok tokens to test it. Welcome to use it!

On January 15, 2026, Google released a new model named TranslateGemma. It can perform translation tasks for 55 languages, including Traditional Chinese and English, based on its specific training. It even supports image input for translation output.

I wondered if it was possible to create an offline "Google Translate-like" web service so that corporate confidential data could be processed in a non-networked environment. Thus, this project was born. This article will introduce TranslateGemma and my personal project.

I. Introduction to TranslateGemma

1. Background & Overview

TranslateGemma is a specialized LLM for translation tasks developed by Google DeepMind, built on the Gemma 3 architecture. It aims to provide the strongest translation capabilities in the open-source community. Unlike general chatbots, TranslateGemma focuses on language conversion. Weights are publicly available on Hugging Face and Vertex AI for local or cloud deployment.

2. Architecture & Training

Its core advantage comes from a unique "two-stage training" process:

Supervised Fine-Tuning (SFT): Utilizing high-quality human translation data and synthetic data generated by Gemini.
Reinforcement Learning (RL): Further guided by translation reward models like MetricX-QE and AutoMQM to align with human preferences and semantic precision.
Model Sizes: Available in 4B (mobile), 12B (laptop/workstation), and 27B (cloud).

3. Key Features

Supported Languages: 55 core languages with training on nearly 500 language pairs. > PS: According to the tech report, Traditional Chinese-related pairs include EN/Cantonese -> Trad. Chinese and Trad. Chinese -> Cantonese.
Multimodal Potential: Inherits Gemma 3's vision capabilities, enabling "visual translation" to understand text context in images (signs, menus, etc.).
High Efficiency: The 12B version often outperforms larger unspecialized models in translation quality. > Reminder: Max Context Window is 2k.

4. Performance & Applications

Excelled in authority benchmarks like WMT24++.

The 12B version's quality (MetricX) even surpasses the general Gemma 3 27B model, proving the effectiveness of specialized training.
Its flexibility makes it the best choice for balancing lightweight design and high quality in the open-source community.

5. Information

Google Blog: https://blog.google/innovation-and-ai/technology/developers-tools/translategemma/
Huggingface: https://huggingface.co/collections/google/translategemma
Tech Report: https://arxiv.org/pdf/2601.09012

II. Open Translate — Modern Interface for TranslateGemma

Open Translate was created to bridge the gap in localization and privacy.

1. Technology Stack

(1) Backend: FastAPI for high-performance async API. Uses Hugging Face transformers to call the translategemma-4b-it model with CUDA acceleration.
(2) Frontend: React (Vite) + Bootstrap for a clean, modern UI with real-time previews.
(3) Database: SQLite (SQLAlchemy) for translation history logs.

2. Highlights & Localization

Multimodal Support: Image translation interface for screenshots, signs, or documents.
Trad. Chinese Optimization: Integrated OpenCC to convert outputs into Taiwan-style phrasing and Traditional Chinese characters.
Privacy & Security: Supports full offline deployment via Docker to ensure data remains internal.

3. Quick Start

Google Colab: One-click script with Node.js/Python setup and ngrok access. https://colab.research.google.com/github/simonliu-ai-product/open-translate/blob/main/open_translate_project_workflow.ipynb
Docker Compose: Single command to run locally on NVIDIA GPUs.

3. Github

Link: https://github.com/simonliu-ai-product/open-translate/tree/main

III. Conclusion

The release of TranslateGemma proves that specialized small models can punch above their weight.

Democratizing Compute: 4B models provide professional quality on home laptops, lowering the barrier to entry.
Multimodal is Future: Translation moves beyond text to direct visual understanding, changing how we interact with the world.
Open Source Value: Combining Google's models with modern web frameworks enables rapid problem-solving.

Open Translate is just a starting point. I will continue to optimize localization and explore integration with AI Agents. Welcome to download the source code on GitHub, give it a Star, or test it via Colab!

I am Simon

Hi everyone, I am Simon Liu, an AI Solutions Expert and a Google Developer Expert (GDE) in GenAI. I look forward to helping enterprises implement AI technologies. If this article was helpful, please give it a "Clap" on Medium and follow my account. Feel free to leave comments on my LinkedIn!

My Personal Website: https://simonliuyuwei.my.canva.site/link-in-bio

[AI Agent] TPU-Based AI Agent Development: Integrating the Twinkle AI Open Source Model (gemma-3–4B-T1-it) with Google ADK Tools

Yu-Wei Simon Liu (Simon Liu) — Mon, 12 Jan 2026 02:44:31 +0000

Special Thanks: I would like to express my gratitude to APMIC and the Twinkle AI community for their assistance, which made the completion of this article possible.

Original Chinese Post: https://medium.com/@simon3458/twinkleai-gemma-3-t1-4b-adk-agent-d3309665f448

As Large Language Model (LLM) technology enters a stage of maturity, the focus of developers has shifted from simple "conversation generation" to "AI Agents" capable of autonomous planning and execution. However, creating an Agent that understands Taiwan's local culture and can accurately execute complex tool calls presents two major challenges: first, general-purpose models often lack understanding of local regulations and context; second, the high cost of GPU computing power limits the widespread adoption of applications.

This article will guide you through exploring: building AI Agent application services by combining the matrix computation advantages of Google TPU, Twinkle AI's gemma-3–4B-T1-it open-source model (optimized specifically for the Taiwanese context), and the Google ADK (Agent Development Kit).

We will start with the underlying architecture of TPUs to explain why they are accelerators for AI inference. Next, we will introduce how Twinkle AI solves "alignment drift" and strengthens Function Calling capabilities. Finally, we will conduct a hands-on walkthrough using Google Colab, stacking and integrating these technologies to build an AI Agent from scratch that is responsive, understands Taiwanese linguistic habits, and can actually query stock information.

I. Introduction to Google TPU

1. What is a TPU (Tensor Processing Unit)?

Google TPU is a "Domain-Specific Architecture" (DSA) integrated circuit tailored specifically for machine learning workloads. Unlike traditional processors that need to handle various general-purpose tasks, the core design of the TPU adopts a "Systolic Array" architecture. This design mimics the way a heart beats, allowing data to flow rhythmically between thousands of arithmetic units within the chip. This architecture enables the TPU to significantly reduce frequent memory access when performing matrix multiplication—the core operation of neural networks—thereby breaking through the "von Neumann bottleneck" of traditional computer architectures and achieving extremely high computational density and efficiency.

2. The Difference Between TPU and GPU

The fundamental difference between the two lies in the philosophical opposition of "Specialization" vs. "Generalization."

GPU (Graphics Processing Unit): essentially a general-purpose parallel processor designed for graphics rendering. It retains a massive amount of control logic and cache to handle complex instruction streams, giving it high flexibility and a powerful CUDA software ecosystem suitable for highly variable research and applications.
TPU: sacrifices generality (it cannot efficiently handle non-matrix operations) and removes hardware units irrelevant to AI, dedicating all released resources to matrix operation units. This gives TPUs higher computational efficiency when processing large-scale static computation graphs in specific formats (such as bfloat16). However, the development barrier is higher than GPUs when dealing with dynamic control flows or custom operators, usually relying on the XLA compiler and JAX framework for optimization.

II. Introduction to the Twinkle AI gemma-3–4B-T1-it Model

(Image Source: Huggingface)

gemma-3-4B-T1-it is a 4B parameter model launched by Twinkle AI based on the Google Gemma 3 architecture. It aims to solve the "alignment drift" problem caused by uneven data in mainstream foundation models and to practice the concept of "Sovereign AI."

The model has been deeply optimized for the Taiwanese context, correcting:

Vocabulary Misuse/Appropriation: (e.g., distinguishing between terms for "quality" and "mass").
Legal and Institutional Hallucinations: (Citing current laws of the Republic of China rather than laws of the PRC).
Cultural Meme Disconnects: (Understanding internet slang from communities like PTT and Dcard).

Through Gemma 3's "Local-Global Hybrid Attention Mechanism" and a 128K token context window, T1–4B-it achieves deep cultural alignment at a lightweight scale, positioning itself as a language model focused on Agent workflows and local needs.

regarding dataset selection and ecosystem collaboration, T1–4B-it adopts a rigorous data strategy. Training data includes lianghsun/tw-reasoning-instruct (reasoning instructions) designed for the Taiwan context, nvidia/Nemotron (instruction following), lianghsun/tw-contract-review-chat (contract review), and Chain of Thought (CoT) data prepared by Kerg (such as tw_mm_R1). We thank APMIC for providing critical computing support for the infrastructure to make this possible.

For architecture designed to strengthen Function Calling, T1–4B-it specifically introduces the Hermes Tool-call Parser format for training, equipping it with powerful Agent capabilities. The model can handle four levels of complex calling scenarios: single function, multiple functions, parallel functions, and parallel multiple functions. In the BFCL evaluation, the model achieved an overall accuracy of 84.5%, with performance on multiple Abstract Syntax Trees (AST) reaching as high as 89%. This demonstrates that at the 4B parameter magnitude, it possesses tool usage and automated execution capabilities surpassing many 7B or 13B models.

Example: Using Google ADK to handle parallel function processing simultaneously.

For detailed information, please visit HuggingFace:

twinkle-ai/gemma-3-4B-T1-it

III. Hands-on: Launching an AI Agent Service on Google Colab via VLLM and Google ADK

If you are not familiar with Google ADK tools, you can read this article first: https://medium.com/@simon3458/google-adk-tools-intro-202504-3181fd6ab567

(Thanks to Twinkle AI community friend Thomas for assisting with the graphics!)

The main goal of this project is to deploy a Twinkle AI Gemma 3 T1 4B model in a Google Colab TPU v5e-1 environment and transform it into an AI Agent capable of executing specific tasks.

Google Colab Notebook Link

1. Environment Preparation and Dependency Installation

Hardware Setup: Confirm the current execution environment is Google TPU v5e-1, hardware designed specifically to accelerate machine learning workloads.

Core Packages: Install the vLLM inference engine that supports TPU acceleration; this is key to making the model run fast. Simultaneously install OpenAI SDK and LiteLLM for subsequent API connection and forwarding.

2. Launching vLLM Inference Service

Load Model: Start the vLLM server via terminal commands and load the Twinkle AI Gemma-3–4B-T1-it model.

Enable Advanced Features: Configure parameters at startup to enable the model's "Auto Tool Choice" and "Hermes Tool Parser," giving the model the ability to understand and call external tools.

Verify Service Status:

Check if the model API is successfully online.
Perform simple conversation tests to confirm the model responds normally.
Critical Test: Test if the model can correctly parse "Function Calling" (e.g., asking about database structures to confirm the model returns the correct tool execution request).

3. Setting up the LiteLLM API Bridge

Configure Forwarding Rules: Create a configuration file to forward standard API requests to the backend vLLM service. This step is to standardize the model interface for compatibility with Google ADK.

Start Proxy Service: Launch the LiteLLM proxy server in the background and monitor it until the service is fully ready (model list appears).

4. Integrating Google ADK (Agent Development Kit)

Get Agent Example: Download a pre-written Stock Query Agent example project from GitHub.

Install Agent Dependencies: Install the Python packages required for the Agent project.

Set Environment Variables: Configure the keys and API addresses needed for the Agent connection, pointing them to the LiteLLM service we just set up.

Run and Test Agent: Launch the Google ADK command-line interface and actually converse with the Agent (e.g., asking for TSMC's stock price). At this point, the Agent will automatically determine the need, call the stock query tool, retrieve data, and finally use the Gemma model to generate a natural language response.

5. (Optional) Establishing a Remote Development Tunnel

Setup ngrok: Use the ngrok tool to expose the API service inside Colab to the public internet.

Local Connection: This allows developers to develop the ADK frontend or logic on their local machine while leaving the heavy model inference computations to run on the TPU in Colab, achieving an efficient "Local Development, Cloud Inference" model.

This process demonstrates the complete integration from underlying model deployment and mid-layer API forwarding to upper-layer Agent application logic, utilizing Google Colab's TPU computing power to build intelligent AI applications.

IV. Conclusion

This hands-on exercise is not only a display of a technology stack but also verifies the huge potential of combining "Specialized Hardware" with "Localized Small Models."

Through the specialized acceleration of Google TPU v5e, we proved that even a lightweight model at the 4B parameter level, when paired with high-quality localized instruction fine-tuning (such as the efforts of Twinkle AI Gemma-3-T1-it) and an appropriate inference framework (vLLM + Google ADK), can demonstrate logical reasoning and tool usage capabilities that transcend its size class.

This solution offers three important insights for developers:

Compute is no longer a high wall: TPUs provide an efficient alternative to GPUs. Through platforms like Colab, developers can access powerful matrix computing resources with a lower barrier to entry.
Localization is crucial: The performance of the Twinkle AI model proves that models which solve "cultural disconnects" and "regulatory hallucinations" are better suited for actual business and life scenarios—an advantage general-purpose models struggle to replace.
Standardization of Agent Development: The introduction of Google ADK and standardized APIs (LiteLLM) evolves Agent development from "hand-crafting Prompts" to modular engineering practices, significantly improving development efficiency and stability.

With the open-sourcing of the Google Gemma 3 architecture and the ubiquity of TPU cloud resources, we are on the eve of a blossoming of AI applications. I hope this tutorial helps developers in various fields quickly build intelligent assistants that understand local languages and solve real problems, truly realizing the democratization and innovation of AI technology.

I am Simon

Hello everyone, I am Simon Liu (Liu Yu-wei), an AI Solutions Expert and currently a Google Developer Expert (AI Role). I look forward to helping enterprises implement Artificial Intelligence technologies to solve problems.

If this article was helpful to you, please give it a clap on Medium and follow my personal account so you can read my future articles at any time. You are welcome to leave comments on my LinkedIn to provide feedback and discuss AI-related topics with me. I look forward to being of help to everyone!

My Personal Website:

https://simonliuyuwei.my.canva.site/link-in-bio