🚀 Build an AI-Powered Test Automation Platform from Scratch

冯键（FENG JIAN） — Mon, 04 Aug 2025 16:47:46 +0000

📌 Introduction

Over the past decade, test automation has evolved significantly — from handwritten scripts, to the Page Object Model, and now to no-code platforms.

But element selection remains the most tedious and brittle part of UI testing.

Frameworks like Selenium, Playwright, or Puppeteer are great at automating actions like .click() or .type(). However, they still rely heavily on manual selectors (XPath or CSS) to locate elements — and that’s where the pain lies.

Thanks to AI — especially large language models (LLMs) — we now have the ability to separate:

What to do (natural language instruction)
from
Where to do it (element selector)

In this tutorial, you’ll build a working solution using:

🧠 Talk2Dom — to convert natural language into precise element selectors
🧪 Selenium — to execute actions in a real browser

This means you can now write a test like:

“Find the login button”

…and watch it execute automatically.

🛠️ Quickstart: One-Click Setup with Docker Compose

After cloning the Talk2Dom repository, everything is pre-configured.

Simply run:

docker compose up -d

This starts:

The Talk2Dom backend (API server)
A database store all projects/tokens

Once up and running, you’re ready to test!

🔁 End-to-End Test Script (`e2e.py`)

import requests
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
from selenium.webdriver.remote.remote_connection import RemoteConnection
import time

# 1. Start Selenium Chrome
driver = webdriver.Remote(
    command_executor=RemoteConnection('http://localhost:4444/wd/hub', resolve_ip=False),
    desired_capabilities=DesiredCapabilities.CHROME
)
driver.get("https://example.com")
time.sleep(2)  # Wait for the page to load
html = driver.page_source

# 2. Call Talk2Dom API to locate the element
headers = {
    "x-api-key": "your_api_key",
    "x-project-id": "your_project_id"
}
resp = requests.post("http://localhost:8000/api/v1/inference/locator", headers=headers, json={
    "instruction": "find the login button",
    "html": html,
    "url": driver.current_url
})
data = resp.json()
selector = data["selector_value"]

# 3. Use Selenium to perform the action
el = driver.find_element(By.CSS_SELECTOR, selector)
el.click()

print("✅ Test completed.")
driver.quit()

🔐 Security Best Practices

✅ Deploy behind an internal firewall (optional)
✅ Limit HTML payload size to prevent injection attacks
✅ Use API key authentication if the service is public

📊 Conclusion

By combining:

Talk2Dom for natural language → selector translation
Selenium for real browser interaction

…we decouple what you want to test from how it’s implemented.

This clean separation leads to a powerful and lightweight AI-assisted test automation workflow.

🔥 Benefits

Boosts developer productivity
Enables QA teams to write tests in plain English
Lays the groundwork for a low-code automation platform

Fine-Tuning Whisper for Japanese-to-Chinese Speech Translation — A Lightweight Approach

冯键（FENG JIAN） — Sun, 15 Jun 2025 18:49:13 +0000

OpenAI’s Whisper is well known for its robust multilingual transcription and English-targeted translation. But what if we want to directly translate Japanese speech into Chinese? In this project, I adapted Whisper’s tiny and base models to perform Japanese-to-Chinese speech translation — a task Whisper doesn’t support out of the box.

🎯 Motivation

Japanese media like anime, drama, and films are hugely popular among Chinese-speaking audiences. However, most existing translation pipelines either route through English or require large GPU resources.

I wanted to explore a low-resource solution that:

Translates directly from Japanese to Chinese
Can run on CPU-only or edge devices

📁 Dataset: ScreenTalk-JA2ZH

To fine-tune Whisper, I created a domain-specific dataset of Japanese audiovisual content with aligned Chinese subtitles.

🎬 Domains: Japanese films, TV dramas, anime
⏱️ Size: 582h train / 73h val / 73h test
📎 Format: 16kHz mono WAV + Simplified Chinese subtitles
✅ Sentence-level alignment, cleaned and manually verified

🔗 A smaller version is publicly available:

👉 ScreenTalk-JA2ZH-XS on Hugging Face

🛠️ Fine-Tuning Setup

Hyperparameter	Value
Epochs	20
Learning rate	3e-4
Precision	fp16
Batch size (tiny/base)	96 / 64
Eval strategy	Step-based
Early stopping	Patience = 5

We fine-tuned both Whisper tiny and Whisper base using the same training pipeline.

📈 Results

🔸 Whisper Tiny

✅ Lightweight, fast
❌ BLEU ≈ 0.60
❌ Prone to overfitting and semantic drift in long/complex speech

🔹 Whisper Base

✅ BLEU = 0.7179
✅ Stronger generalization and fluency
✅ Suitable for CPU deployment (edge ready)

👉 BLEU scores steadily improved even when token-level loss increased — highlighting that loss is not always a good proxy for translation quality.

🤔 Key Takeaways

Whisper can be adapted for non-English language pairs
Domain-specific data (like anime or TV) greatly improves model performance
Model capacity matters: Tiny is efficient but not enough for expressive, noisy domains
BLEU is limited — future work should include COMET, chrF, or human evals

🔮 What’s Next?

Fine-tune larger Whisper models (medium, large)
Try LoRA or other parameter-efficient tuning techniques
Expand dataset to cover conversational, technical, and news speech

🚀 Try It Out

🧠 Models available on Hugging Face:

Thanks to the open-source Whisper community and everyone working to break language barriers with AI.

👉 Follow me for more multilingual AI experiments!

DEV Community: 冯键（FENG JIAN）