DEV Community

Cover image for How I built my open source testing agent..🤖
robin gupta
robin gupta

Posted on

How I built my open source testing agent..🤖

Hercules is an open-source, AI-powered testing agent designed to automate end-to-end web application testing using Gherkin feature files. This guide provides a technical overview of Hercules' architecture, functionality, and usage, omitting marketing jargon.

Core Concepts

Hercules operates on a multi-agent arca fewhitecture based on the AutoGen/AG2 framework. It employs a few sub-agents:

  1. Planner Agent: Responsible for interpreting Gherkin steps and formulating an execution plan.
  2. Browser Navigation Agent: Executes the plan by interacting with the web browser using a library of specialized tools.

The Tools Library is central to Hercules' functionality. These tools could be:

  • Sensing Tools: Gather information about the web page state (e.g., geturl, get_dom_with_content_type).
  • Action Tools: Perform interactions with the web page (e.g., click, enter_text, openurl).

Tools return natural language descriptions of their outcomes, facilitating communication with the LLM.

DOM Distillation

To improve efficiency, Hercules employs DOM Distillation, a process that simplifies the HTML DOM into a more manageable JSON representation. This distilled DOM focuses on relevant elements, reducing the cognitive load on the LLM.

Hercules utilizes the DOM Accessibility Tree for distillation, aligning with the needs of web automation.

Architecture

Hercules' architecture can be visualized as follows:

Architecture diagram

The Planner Agent uses an LLM (e.g., GPT-4o) to interpret the Gherkin steps and create an execution plan. The Browser Navigation Agent then uses this plan to execute the test steps by leveraging the tools library to interact with the web browser.

Installation and Usage

Hercules offers several installation methods:

  • PyPI: pip install testzeus-hercules followed by playwright install --with-deps.
  • Docker: docker pull testzeus/hercules:latest. See the README for details on mounting volumes and setting environment variables.
  • Source Code: Clone the repository and use the provided make commands.
  • Helper Script: Use the provided helper_script_custom.sh for automated setup on Linux/macOS.

Running tests requires specifying parameters such as the input feature file, output path, and test data path. These can be provided directly or via environment variables when using Docker.

The agents_llm_config.json file allows for advanced configuration of LLMs used by the agents.

Supported LLMs and Platforms

Hercules supports various LLMs, including OpenAI, Anthropic, Groq, and Mistral. It also integrates with cloud platforms like AWS Bedrock, GCP VertexAI, and AzureAI, as well as browser providers like BrowserStack, LambdaTest, BrowserBase, and AnchorBrowser.

Extending Hercules

Developers can extend Hercules by creating custom tools. Tools are defined as Python functions decorated with @tool. The decorator should include a clear description and parameter annotations to facilitate LLM interaction. Custom tools can be added by specifying their directory path using the ADDITIONAL_TOOL_DIRS environment variable.

Key Features

  • Gherkin-based testing: Simplified test case creation and management.
  • No-code automation: Reduces the need for complex scripting.
  • Salesforce support: Handles complex UIs and interactions.
  • Multilingual capabilities: Supports testing in multiple languages.
  • CI/CD integration: Facilitates automated testing within development pipelines.
  • Mobile device emulation: Enables testing on various mobile devices.
  • Security and accessibility testing: Integrates with Nuclei for security scanning and supports WCAG standards for accessibility testing.

Conclusion

Hercules provides a novel approach to web application testing by leveraging the power of AI and a multi-agent architecture. Its open-source nature, combined with its extensibility and support for various platforms and LLMs, makes it a valuable tool for developers seeking to streamline and enhance their testing processes.

If you like the article, please feel free to try out Hercules and Star the repo here: https://github.com/test-zeus-ai/testzeus-hercules/.

Sentry image

Hands-on debugging session: instrument, monitor, and fix

Join Lazar for a hands-on session where you’ll build it, break it, debug it, and fix it. You’ll set up Sentry, track errors, use Session Replay and Tracing, and leverage some good ol’ AI to find and fix issues fast.

RSVP here →

Top comments (0)

Billboard image

The Next Generation Developer Platform

Coherence is the first Platform-as-a-Service you can control. Unlike "black-box" platforms that are opinionated about the infra you can deploy, Coherence is powered by CNC, the open-source IaC framework, which offers limitless customization.

Learn more

👋 Kindness is contagious

Please leave a ❤️ or a friendly comment on this post if you found it helpful!

Okay