DEV Community: robin gupta

Open source stack for agentic testing using Deepseek

robin gupta — Tue, 28 Jan 2025 13:22:43 +0000

At TestZeus, we're pushing the boundaries of open source software testing stack by combining the power of innovations with cutting-edge AI technologies. Today, we’re excited to share how you can leverage Deepseek R1 for generating both UI and API tests, and seamlessly execute them using Hercules, the world’s first open-source testing agent.

The Open-Source Advantage

Open-source tools have revolutionized the software industry, offering transparency, flexibility, and significant cost savings. Deepseek R1, released under the permissive MIT license, harnesses advanced AI capabilities to generate high-quality test cases for both UI and API scenarios. With its Mixture of Experts (MoE) architecture, Deepseek R1 excels in logical inference, mathematical problem-solving, and real-time decision-making, ensuring comprehensive and effective test generation.

Hercules, our groundbreaking open-source testing agent, empowers teams to execute these tests efficiently, delivering unparalleled automation capabilities. By leveraging these tools, you not only accelerate your testing process by up to 70-80%, but also reduce dependencies on proprietary solutions that can come with hidden costs and security vulnerabilities.

Example Prompts for Test Generation with Deepseek R1
Deepseek R1 simplifies test generation by using natural language prompts. Here are a few examples to get you started:

Generating UI Tests:
Prompt: Act as an expert QA Analyst with deep expertise in Behavior-Driven Development (BDD) and Gherkin syntax. Your task is to analyze the provided **Functional Requirement Document (FRD)** and generate comprehensive **positive and negative functional test scenarios** in Gherkin format. Follow these guidelines: 1. **Input Processing**: - Parse the FRD to identify **all functional requirements**, including user stories, acceptance criteria, edge cases, and error-handling rules. - Extract preconditions, user actions, system responses, and postconditions. 2. **Test Scenario Generation**: - For each requirement, generate **at least 1 positive test** and **2–3 negative tests** covering diverse failure modes. 3. **Gherkin Structure**: - Use theFeature,Scenario,Given,When,Thensyntax. - Include a **descriptive title** and **purpose** for each test. - Use **data tables** and **examples** where applicable for parameterization. - Ensure scenarios are atomic, independent, and executable. 4. **Output Format**:gherkin Scenario: <Positive Test Title> Given <precondition> When <action> Then <expected outcome> Feature: <Feature Name> Scenario: <Negative Test Title> Given <precondition> When <invalid/edge-case action> Then <system should handle error with [specific message/behavior]>5. **Examples**: - *FRD Requirement*: "User must log in with a valid email and password." - *Generated Tests*:gherkin Scenario: Successful login with valid credentials Given the user is on the homepage When they click on the search icon And enter the "search term" Then they should be shown the relevant search results. Feature: User Login Scenario: Login attempt with invalid password Given the user is on the login page When they enter "test@example.com" and "WrongPass" And click the "Login" button Then the system should display "Invalid credentials" And the user remains on the login page6. **Validation**: - Ensure all FRD requirements are mapped to tests. - Avoid redundancy; prioritize clarity and coverage. - Include error messages, boundary values, and security checks for negative tests. Return **only the Gherkin code**, organized by feature and test type, with no additional commentary."
Generating API Tests:
Prompt: Create API test cases for the attached OpenAPI Spec. Test scenarios should be in Gherkin format, to be consumed in a REST client.

These prompts unleash Deepseek R1’s ability to create structured, detailed, and executable test cases for both UI and API layers.

Sample Gherkin Tests Generated by Deepseek R1
UI Test Case:
`Feature: Login functionality testing
Scenario: Successful login with valid credentials
Given the user is on the login page
When the user enters valid credentials
And clicks the login button
Then the user should be redirected to the dashboard

Scenario: Unsuccessful login with invalid credentials
Given the user is on the login page
When the user enters invalid credentials
And clicks the login button
Then an error message "Invalid credentials" should be displayed`

API Test Case:
`Feature: Payment Gateway API Testing
Scenario: Successful payment
Given the payment gateway is active
When a valid payment request is sent
Then the response code should be 200
And the response body should contain "payment_status: success"

Scenario: Payment with invalid card details
Given the payment gateway is active
When a payment request is sent with invalid card details
Then the response code should be 400
And the response body should contain "error: Invalid card details"`

Running Tests with Hercules
Hercules makes executing these Gherkin-based tests effortless. Here’s a quick primer:

Step 1: Setup Hercules
Clone the open-source Hercules repository from GitHub and set up the environment by following the installation guide.

Step 2: Prepare Your Test Files
Save the Gherkin test cases into .feature files and place them in the specified directory.

Step 3: Execute Tests
Run the following command to execute your tests:

testzeus-hercules --input-file opt/input/test.feature --output-path opt/output --test-data-path opt/test_data --llm-model gpt-4o --llm-model-api-key sk-proj-k.......
Step 4: View Results
Hercules generates detailed execution logs and test reports, making it easy to review the outcomes and identify any issues.

Cost Savings and Security Benefits of an Open-Source Stack
Lower Costs: By using open-source tools like Deepseek R1 and Hercules, you eliminate licensing fees and significantly reduce your testing expenses.

Enhanced Security: Open-source tools are transparent and community-vetted, enabling quicker identification and resolution of vulnerabilities. Also, you can deploy them in an air gapped fashion at your end.

Flexibility and Customization: Tailor the tools to fit your specific testing needs, avoiding the one-size-fits-all limitations of proprietary software.

Community Support: Benefit from a vibrant community of developers and contributors who constantly improve the tools.

Conclusion
Deepseek R1 and Hercules represent the future of software testing by combining AI-driven test generation with robust, open-source test execution capabilities. Deepseek R1’s advanced reasoning and problem-solving abilities ensure comprehensive test coverage, while Hercules provides an efficient and effective test automation solution. You can find the full video here: Youtube

Whether you're a small team or a large enterprise, this powerful duo can help you achieve higher efficiency, lower costs, and improved software quality. Ready to transform your testing strategy? Join the open-source revolution with TestZeus, and experience the power of intelligent, efficient, and affordable software testing.
Check out the open source solution here, and drop us a star or a line: World's first open source testing agent

How I built my open source testing agent..🤖

robin gupta — Mon, 27 Jan 2025 19:28:52 +0000

Hercules is an open-source, AI-powered testing agent designed to automate end-to-end web application testing using Gherkin feature files. This guide provides a technical overview of Hercules' architecture, functionality, and usage, omitting marketing jargon.

Core Concepts

Hercules operates on a multi-agent arca fewhitecture based on the AutoGen/AG2 framework. It employs a few sub-agents:

Planner Agent: Responsible for interpreting Gherkin steps and formulating an execution plan.
Browser Navigation Agent: Executes the plan by interacting with the web browser using a library of specialized tools.

The Tools Library is central to Hercules' functionality. These tools could be:

Sensing Tools: Gather information about the web page state (e.g., geturl, get_dom_with_content_type).
Action Tools: Perform interactions with the web page (e.g., click, enter_text, openurl).

Tools return natural language descriptions of their outcomes, facilitating communication with the LLM.

DOM Distillation

To improve efficiency, Hercules employs DOM Distillation, a process that simplifies the HTML DOM into a more manageable JSON representation. This distilled DOM focuses on relevant elements, reducing the cognitive load on the LLM.

Hercules utilizes the DOM Accessibility Tree for distillation, aligning with the needs of web automation.

Architecture

Hercules' architecture can be visualized as follows:

The Planner Agent uses an LLM (e.g., GPT-4o) to interpret the Gherkin steps and create an execution plan. The Browser Navigation Agent then uses this plan to execute the test steps by leveraging the tools library to interact with the web browser.

Installation and Usage

Hercules offers several installation methods:

PyPI: pip install testzeus-hercules followed by playwright install --with-deps.
Docker: docker pull testzeus/hercules:latest. See the README for details on mounting volumes and setting environment variables.
Source Code: Clone the repository and use the provided make commands.
Helper Script: Use the provided helper_script_custom.sh for automated setup on Linux/macOS.

Running tests requires specifying parameters such as the input feature file, output path, and test data path. These can be provided directly or via environment variables when using Docker.

The agents_llm_config.json file allows for advanced configuration of LLMs used by the agents.

Supported LLMs and Platforms

Hercules supports various LLMs, including OpenAI, Anthropic, Groq, and Mistral. It also integrates with cloud platforms like AWS Bedrock, GCP VertexAI, and AzureAI, as well as browser providers like BrowserStack, LambdaTest, BrowserBase, and AnchorBrowser.

Extending Hercules

Developers can extend Hercules by creating custom tools. Tools are defined as Python functions decorated with @tool. The decorator should include a clear description and parameter annotations to facilitate LLM interaction. Custom tools can be added by specifying their directory path using the ADDITIONAL_TOOL_DIRS environment variable.

Key Features

Gherkin-based testing: Simplified test case creation and management.
No-code automation: Reduces the need for complex scripting.
Salesforce support: Handles complex UIs and interactions.
Multilingual capabilities: Supports testing in multiple languages.
CI/CD integration: Facilitates automated testing within development pipelines.
Mobile device emulation: Enables testing on various mobile devices.
Security and accessibility testing: Integrates with Nuclei for security scanning and supports WCAG standards for accessibility testing.

Conclusion

Hercules provides a novel approach to web application testing by leveraging the power of AI and a multi-agent architecture. Its open-source nature, combined with its extensibility and support for various platforms and LLMs, makes it a valuable tool for developers seeking to streamline and enhance their testing processes.

If you like the article, please feel free to try out Hercules and Star the repo here: https://github.com/test-zeus-ai/testzeus-hercules/.

Making the Web More Inclusive: Accessibility Testing with open source testing agents

robin gupta — Tue, 24 Dec 2024 16:03:19 +0000

Hey dev community! With the holidays right around the corner, it feels like the perfect time to talk about one of the best gifts we can give our users: an accessible web. Here at TestZeus, we’ve been hard at work adding Accessibility Testing to our open-source Hercules Agent—and I’m super excited to share the details with you!

Why Accessibility Matters

Accessibility is about making sure everyone can use your website or app, regardless of ability. That includes users who rely on screen readers, have motor difficulties, or experience color blindness, among many other scenarios. It’s not just a moral responsibility or a legal checkmark—it’s about crafting better user experiences for everyone.

Meet Hercules

If you haven’t heard of it yet, Hercules is our open-source testing agent that automates various aspects of software testing. By adding Accessibility Testing, we aim to reduce the friction developers often face when trying to meet WCAG (Web Content Accessibility Guidelines). Some highlights:

Covers WCAG 2.0, 2.1, and 2.2
Costs less than $1 per scan (so, super accessible for smaller teams too!)
Open-source and easy to integrate into your CI/CD pipeline

In other words, we’re focusing on practicality and affordability, so you don’t need a huge budget or specialized skillset to make your site more inclusive.

Quick Setup

Clone the Repo
- Head over to our GitHub repo and clone it.
Install Dependencies
- A quick 'pip install testzeus-hercules'
Write Your Test in Gherkin
- Hercules uses Gherkin-like .feature files. It’s straightforward and keeps the logic clear.
Run Hercules
- A single command to scan for accessibility issues. Reports are automatically generated, so you can jump right into fixing any violations.

Sample Test Scenario

We’ve included a snippet below to show how a typical .feature file might look, using the Zerodha brokerage calculator page as an example:

Feature: Check accessibility for brokerage calculator on Zerodha website
  # This feature tests the brokerage calculator, and contract notes on Zerodha website

  Scenario Outline: Check calculations on brokerage and SEBI charges
    Given the user is on "https://zerodha.com/brokerage-calculator#tab-equities"
    Then validate the page for accessibility.
    Then there should not be any issues.

Here’s the breakdown:

Feature: Provides context—testing Zerodha’s brokerage calculator.
Scenario Outline: Explains the specific scenario.
Steps:
- Given: Navigate to a specific URL.
- Then: Validate accessibility.
- Then: Ensure zero issues remain (fingers crossed!).

Hercules interprets this scenario, visits the page, scans it, and flags any WCAG non-compliances.

Viewing Your Results

Once your tests complete, Hercules shows you a report detailing any accessibility issues found:

Issue Description: A quick summary of the violation.
WCAG Reference: Which guideline is being broken.
Severity: Ranges from minor to critical.
Location: Points you to the exact DOM element or area in the UI.

This immediate feedback loop makes it easy for devs to fix issues before they reach production.

Integrating with CI/CD

Because Hercules is open-source and script-friendly, you can plug it into your CI pipeline. For example, in GitHub Actions, you can create a workflow that runs Hercules every time you push new code. That way, you’ll catch accessibility regressions early in the development process.

Going Beyond Automated Checks

While Hercules does a fantastic job with automated scans, keep in mind that accessibility is a multifaceted area. You might still want to:

Test with screen readers like NVDA or VoiceOver.
Check color contrasts in real-world scenarios.
Solicit feedback from users with disabilities for real-life insights.

Automated testing is a huge help, but manual testing still plays a key role in thorough accessibility coverage.

Wrapping Up

This holiday season, consider giving the gift of an accessible experience to your users. With Hercules, you can weave accessibility checks right into your development cycle—reducing barriers and making the web a more inclusive place.

Here are some helpful links:

Repo: TestZeus Hercules GitHub
Latest Release: Version 0.0.16
Demo Video: Check it out on YouTube

If you have any questions, feedback, or ideas for improvement, we’d love to hear from you! Let’s work together to make the internet more welcoming for all.

Happy Holidays and Happy Testing!

Hercules⚡: The Testing agent you always wanted

robin gupta — Sat, 07 Dec 2024 20:51:22 +0000

If you've ever spent hours wrestling with flaky tests, dreaded the maintenance burden, or felt limited by tools that don’t keep up with your needs, you’re not alone.

As developers, QA engineers, and product owners, we know this struggle all too well.

That's why we created Hercules 🛡️—

The first open-source testing agent that’s flexible, intelligent, and powerful.

🚀 What is Hercules?

Hercules is an end-to-end testing agent that takes care of the heavy lifting so you can focus on building great software.

It simplifies testing across APIs, browsers, and databases—no scripting, no maintenance headaches, and no costs.

Think of Hercules as your digital testing assistant that:

✅ Reads your Gherkin files (or structured test cases)

✅ Executes tests seamlessly across tools like APIs, web browsers, and databases

✅ Analyzes results to give you clear, actionable insights

🔑 Key Features

1️⃣ Supports API, Web, and Database Testing 📊

Test your APIs (REST, GraphQL) with ease
Run browser-based UI tests with minimal configuration
Verify database queries and responses effortlessly

2️⃣ No-Code Simplicity 🚫💻

Write tests in Gherkin syntax—simple, human-readable, and powerful
Hercules parses, executes, and reports results—no need for coding expertise.

3️⃣ Open-Source and Free 🔓

Fully open-source under the MIT License
Customize it to your needs, or extend it as you grow

4️⃣ Agent-Driven Testing 🤖

Hercules is not just another tool—it’s an AI-powered agent designed to perform, adapt, and deliver reliable results every time.

🌐 How Does It Work?

Write Your Tests

Create tests in Gherkin format (Given-When-Then)—clean, simple, and intuitive.
Run Tests

Hercules runs your tests across APIs, browsers, or databases seamlessly.
View Results

Get structured and insightful reports that help you identify issues quickly.

Example:

Feature: API Response Validation  
  Scenario: Verify API returns 200 status code  
    Given the API endpoint "https://api.example.com/health"  
    When I send a GET request  
    Then the response status should be 200

🎯 Why Hercules?

We built Hercules to solve real testing pain points:

No-Code Simplicity 🧩: Use Gherkin—not scripts
Time-Saving ⏳: Minimal setup, minimal maintenance
Cross-Tool Execution ⚙️: API, web, database—all in one place
Cost-Free and Customizable 💰: Fully open-source and flexible

Testing shouldn't slow you down. It should empower you.

With Hercules, testing becomes as seamless and reliable as the rest of your development pipeline.

🌟 Get Started with Hercules

It’s easy to start testing with Hercules:

👉 Explore the GitHub Repository

👉 Clone it, run it, and experience end-to-end testing that just works!

💡 We Need Your Feedback

Hercules is for the community, by the community. We’d love for you to try it, break it, and improve it!

Found a bug? Open an issue.
Got a feature request? Let us know.
Want to contribute? PRs are welcome!

🚀 Simplify Your Testing. Today.

Join us on this journey to make testing smarter, faster, and frustration-free. Hercules is open-source, powerful, and ready for you.

🔗 Start Testing with Hercules Now

⚡ #OpenSource #Testing #Automation #NoCode #DevTools #Hercules

Let me know if you'd like me to adjust further or highlight specific sections. 🚀