DEV Community: Raja

AI-Assisted Testing with Browseruse

Raja — Sun, 06 Apr 2025 09:22:14 +0000

Why High-Level AI-Powered Testing?

Automated UI testing is a critical part of modern QA. However, writing stable UI tests—especially for legacy applications or hard-to-automate interfaces—can be:

Time-consuming
Fragile due to frequent DOM changes or lack of test IDs
Manual and resource-intensive to maintain

AI-powered high-level automation offers a compelling alternative:

Reduces QA effort for sanity and smoke tests
Avoids the need for writing detailed selectors or test data
Enables automation of legacy or dynamically rendered applications

This blog demonstrates how existing open-source tools can be combined to build flexible and AI-enhanced automation workflows—without relying on expensive commercial platforms.

Tools Used

This approach integrates:

Browseruse: An LLM-driven browser controller that uses Playwright under the hood
Playwright: A robust framework for browser automation
LLMs (Gemini): Handles interpretation of natural language QA tasks

In this example, we use Gemini 2.0 Flash via LangChain due to its free availability. However, you can easily swap in alternatives like GPT-4, Claude 3, or even local models via Ollama.

Available Alternatives in the Market

Several commercial tools offer similar AI-powered testing capabilities, but What differentiates Browseruse + LLM:

Fully open source agent
Compatible with any LLM provider
Easy to extend with custom component logic

Real Example: One Script, No Test Data, No Selectors

Consider a test application running locally at http://localhost:5173/, which includes a form submission page.

Below is a complete script to test it using Browseruse and Gemini:

from browser_use import Agent
from dotenv import load_dotenv
from langchain_google_genai import ChatGoogleGenerativeAI
import asyncio

llm = ChatGoogleGenerativeAI(model="gemini-2.0-flash")
load_dotenv()

async def main():
    success_task = """
    You are an automated QA assistant performing exploratory UI testing.
    Open http://localhost:5173/
    Fill the form with appropriate values
    Click Save after filling mandatory values
    Return task as success if you see success toast message or fail with reasons needed.
    """

    agent = Agent(
        task=success_task,
        llm=llm,
    )

    await agent.run()

asyncio.run(main())

This script requires:

No test-specific data
No HTML selectors or test IDs
No manual handling of dropdowns or wait conditions

The LLM interprets the flow and executes it end-to-end using intelligent heuristics.

You can also write a negative test case to validate failed submissions:

failure_task = """
You are an automated QA assistant performing exploratory UI testing.
Open http://localhost:5173/
Validate if the form works for mandatory values
Return task as success if you see error toast message for validation fails.
"""

Challenges and Considerations

Prompt Design

Complex UI components like Ant Design’s rc-select or dynamic dropdowns require detailed prompt crafting or custom handlers.

Execution Time

LLM-based test runs tend to be slower than hand-coded automation scripts, particularly for long forms or multi-step flows.

Cost

While Gemini Flash is free, using models like GPT-4 or Claude 3 may incur usage-based costs.

Benefits

Using this approach, you can:

Test new UIs without creating test scripts
Add smoke checks for legacy flows without deep DOM knowledge
Detect form-level bugs early in the development cycle

All with the robustness of Playwright and the intelligence of LLMs.

What’s Next

Implement screenshot comparisons for visual regressions
Generate test reports linked to user stories
Auto-record successful runs as replayable test cases
Fine-tune prompts for app-specific language and behavior

Automating Node.js Dependency Upgrades and Build Error Resolution Using AI

Raja — Wed, 01 Jan 2025 13:37:56 +0000

Keeping your Node.js project up to date is crucial to ensure you're leveraging the latest features, security patches, and performance improvements. However, maintaining dependencies and dealing with breaking changes can often feel like a tedious, error-prone task. Wouldn’t it be great if there were a way to automate some of these steps and even get AI-powered suggestions for how to fix issues that arise?

This blog introduces a Python-based script that helps streamline two key aspects of Node.js development: upgrading dependencies and resolving build errors. While this approach may not be the ultimate, fully automated solution, it offers a practical starting point for easing the work involved. The next steps could involve integrating this into your CI/CD pipeline as a bot that creates pull requests (PRs) with the latest dependency upgrades and suggestions for fixing code issues.

Moreover, there’s potential to take this even further—imagine using a specialized AI model that doesn’t just suggest fixes but directly applies them and creates the pull request on your behalf. In this post, we'll explore the current solution and discuss the possible next-level enhancements.

Additionally, while tools like Dependabot already automate dependency updates, this solution offers something a bit different: it doesn’t stop at upgrading libraries—it helps you deal with the consequences of those upgrades by offering suggestions for fixing build errors, which is an area where Dependabot falls short. Let's dive in!

Key Features of the Script

Automated Dependency Upgrades

The script fetches the latest versions of outdated dependencies in your Node.js project and updates the package.json file, similar to what tools like Dependabot do, but with an added focus on analyzing and fixing the consequences of those updates.
Automated Build Process

After upgrading dependencies, the script runs the build process and checks for errors. If the build fails, it logs the error details and attempts to analyze them.
AI-Powered Error Resolution

Once the errors are captured, the script uses generative AI models (like Google Gemini or local models like CodeLlama) to analyze the errors and suggest potential fixes, reducing the burden of debugging.

Now, let’s take a look at how each part of the script works.

1. Upgrading Dependencies

Tools like Dependabot can automatically create pull requests for dependency updates in your repository. However, they only address the upgrade part—they don't deal with the potential breaking changes that can occur when dependencies are updated. This script goes a step further by automating the upgrade of outdated dependencies and allowing you to check for build issues afterwards.

def upgrade_dependencies(project_dir):
    try:
        # Get outdated packages in JSON format
        result = subprocess.run(
            ["npm", "outdated", "--json"],
            cwd=project_dir,
            capture_output=True,
            text=True
        )

        outdated_packages = json.loads(result.stdout)

        # Update package.json with the latest versions
        with open(f"{project_dir}/package.json", "r") as f:
            package_json = json.load(f)

        for package_name, package_info in outdated_packages.items():
            if package_name in package_json.get("dependencies", {}):
                package_json["dependencies"][package_name] = package_info["latest"]

        # Write updated package.json
        with open(f"{project_dir}/package.json", "w") as f:
            json.dump(package_json, f, indent=2)

        # Install updated packages
        subprocess.run(["npm", "install"], cwd=project_dir, check=True)

        return True
    except Exception as e:
        print(f"Error upgrading dependencies: {e}")
        return False

What it does:

The function runs npm outdated --json to fetch outdated dependencies and updates the package.json file with the latest versions. Then, it runs npm install to install those updated packages.
How it's different from Dependabot:

While Dependabot handles the "easy" part of keeping dependencies updated, it doesn't take into account the real-world impact of these updates on your build process. This script not only upgrades the dependencies but also checks whether the upgrade introduces build errors.

2. Handling Build Errors

After upgrading the dependencies, it’s time to build the project. Unfortunately, dependencies sometimes come with breaking changes, and the build may fail. In such cases, error logs are critical for identifying and fixing the issues. This script handles that by logging the errors and running an analysis on them.

def build_project(project_dir):
    try:
        build_result = subprocess.run(
            ["npm", "run", "build"],
            cwd=project_dir,
            capture_output=True, 
            text=True
        )

        if build_result.returncode == 0:
            print("Build successful!")
            return False
        else:
            build_errors = build_result.stdout
            print("Build failed! Errors:")
            print(build_errors)

            with open(f"{project_dir}/build_errors.log", "w") as f:
                f.write(build_errors)

            return True

    except Exception as e:
        print(f"Error building project: {e}")
        return False

What it does:

It runs npm run build and captures any errors. If the build fails, it saves the error logs to a file for further analysis.
How it helps:

After an upgrade, build errors are inevitable. By logging and analyzing them, you can quickly identify where the issue lies and take action accordingly. This function could be extended to integrate directly into a CI/CD pipeline, automating the entire process of upgrading dependencies, building the project, and logging the errors.

3. AI-Powered Error Resolution

The most exciting part of this script is its ability to use AI to suggest fixes for build errors. By using generative AI models, the script attempts to analyze the errors in the build logs and offer practical solutions.

def analyze_build_errors(error_log, project_dir):
    try:
        with open(error_log, "r") as f:
            errors = f.read()

        # Load an open-source AI model (e.g., CodeLlama)
        model_name = "codellama/CodeLlama-7b-hf"
        model = AutoModelForCausalLM.from_pretrained(model_name)
        tokenizer = AutoTokenizer.from_pretrained(model_name)

        suggestions = []
        for error in errors.splitlines():
            if 'error' in error:
                code_snippet = get_code_snippet_around_error(project_dir, error)
                prompt = f"""
                **Error:** {error}
                **Code Snippet:**
                ```
{% endraw %}
typescript
                {code_snippet}
{% raw %}

                ```
                **Instruction:** How can I resolve this error? 
                """

                inputs = tokenizer(prompt, return_tensors="pt")
                input_ids = inputs["input_ids"]
                attention_mask = inputs["attention_mask"]

                output = model.generate(
                    input_ids=input_ids, 
                    attention_mask=attention_mask, 
                    max_new_tokens=100,  
                    num_beams=1,       
                    do_sample=True, 
                    temperature=0.1,
                    top_p=0.9,
                )

                suggestion = tokenizer.decode(output[0], skip_special_tokens=True)
                suggestions.append(suggestion)

        return suggestions
    except Exception as e:
        print(f"Error analyzing build errors: {e}")
        return []

What it does:

This function takes the error logs and uses an AI model to generate possible fixes based on the errors. It pulls relevant code snippets from the project to give the AI context and provides a more accurate suggestion.
How it's different from Dependabot:

Dependabot is excellent at upgrading dependencies automatically, but it doesn’t offer any insights or solutions if the upgrade causes issues in your code. This script goes a step further by offering context-specific suggestions on how to fix those issues, using AI-powered code analysis.

Next Steps: Moving Towards Full Automation

While this script helps automate some of the more manual aspects of dependency management and error resolution, it’s still just a starting point. The next steps could include:

CI/CD Pipeline Integration:

Imagine integrating this process into your CI/CD pipeline as a bot that automatically opens a pull request whenever a dependency upgrade is detected. The bot could include suggested fixes for any issues caused by those upgrades, reducing the manual intervention required.
AI-Driven Code Fixing:

Taking things even further, specialized AI models could not only suggest fixes but also apply them directly to your codebase. The AI could run a full analysis of the errors, apply the necessary code modifications, and then create a pull request on your behalf.

Conclusion

Automating dependency upgrades and build error resolution using AI is an exciting direction for improving Node.js project maintenance. While tools like Dependabot can handle the initial dependency update process, they fall short when it comes to managing the complex consequences of those updates. This script bridges that gap by providing automatic upgrades, build error detection, and AI-powered suggestions for fixes.

Though this is just a starting point, it demonstrates the potential for fully automating these tasks and integrating them into your development workflow. Future iterations could take this approach to the next level by incorporating it into CI/CD pipelines and leveraging more sophisticated AI models to directly fix code and create pull requests.

If you're looking to streamline your Node.js project maintenance, this could be a great place to start. What do you think? How would you improve this idea?

Github reference