DEV Community: gusdmoreira

Beyond the Green Test: A Pragmatic Framework for Evaluating AI Code Quality

gusdmoreira — Tue, 06 Oct 2026 15:34:15 +0000

You prompt your favorite AI coding assistant: "Write a service that processes bulk user telemetry, validates incoming records, and handles batch publishing."

Within seconds, pristine code streams across your screen. The structure looks clean, the syntax leverages modern language features, and it even includes neat documentation comments. You drop it into your repository, run your test suite, and watch the console turn bright green.

Ship it, right?

Not so fast.

While tools like GitHub Copilot, Claude, and specialized coding agents are incredible velocity boosters, they operate on probability, not deep contextual understanding. They excel at the happy path but frequently gloss over the messy realities of production systems: concurrency nuances, resource management under heavy load, security implications, and subtle domain edge cases.

If you blindly merge AI-generated code without a systematic evaluation framework, you are essentially outsourcing your technical debt to a non-deterministic generator.

Let's look at a practical, robust approach to auditing and validating AI-generated code, using Java for our concrete examples.

1. Shift-Left Validation: Define the Contract via TDD

The biggest trap developers fall into is letting the AI write both the implementation and the expectations. If the model hallucinates a requirement or misunderstands a business rule, its generated unit tests will happily validate its own flawed logic.

Instead, practice AI-Driven TDD: Write the test contract yourself before invoking the assistant.

Imagine you need a thread-safe rate limiter service. Before asking the AI to write the class, write your strict unit test suite covering concurrent threads, edge limits, and time windows. For instance, in Java using JUnit 5 and AssertJ:

package com.example.ratelimiter;

import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
import java.time.Duration;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.atomic.AtomicInteger;

import static org.assertj.core.api.Assertions.assertThat;

class RateLimiterServiceTest {

    @Test
    @DisplayName("Should allow requests within limit and throttle excess under concurrency")
    void testConcurrentRateLimiting() throws InterruptedException {
        RateLimiterService limiter = new RateLimiterService(5, Duration.ofSeconds(1));
        int totalThreads = 20;
        ExecutorService executor = Executors.newFixedThreadPool(totalThreads);
        CountDownLatch latch = new CountDownLatch(totalThreads);
        AtomicInteger allowedCount = new AtomicInteger(0);

        for (int i = 0; i < totalThreads; i++) {
            executor.submit(() -> {
                try {
                    if (limiter.tryAcquire("user-123")) {
                        allowedCount.incrementAndGet();
                    }
                } finally {
                    latch.countDown();
                }
            });
        }

        latch.await();
        executor.shutdown();

        // Exactly 5 should pass out of 20 concurrent attempts
        assertThat(allowedCount.get()).isEqualTo(5);
    }
}

The Danger of the "Fake Contract" (Make Your Test Bleed First)
Writing the contract yourself prevents the AI from validating its own flaws, but it opens the door to a quieter issue: you can write a test that passes for the wrong reasons.

For example, if you ask an AI for a deduplication function and write an assertion checking only that the output is a subset of the original elements, that test will happily pass even if the model returns the list completely untouched with duplicates intact. A test that has never failed is not a guard—it is a comment that runs.

The Golden Rule: Before you trust your test suite, break the implementation on purpose (return a dummy value or an empty list) and watch it go red. Pre-fix red is the only real evidence that a test is actually capable of failing. If a known-bad build still passes your test, your contract is an illusion.

By establishing this rigorous contract upfront, you give the AI a rigid mathematical boundary. When it returns the implementation, your test suite acts as an objective security guard.

2. Curate a Domain-Specific "Golden Dataset"

For complex data transformations, parsers, or validation rules, unit tests aren't always enough to catch structural discrepancies. This is where you build a Golden Dataset, a version-controlled reference collection containing edge cases specific to your domain.

You can maintain this as a resource file (like JSON) inside your project structure:

[
  { "inputUserId": "USR-001", "payload": "{\"tier\": \"PREMIUM\", \"amount\": 1500.00}", "expectedStatus": "ACCEPTED" },
  { "inputUserId": "USR-002", "payload": "MALFORMED_JSON_STRING", "expectedStatus": "REJECTED_MALFORMED" },
  { "inputUserId": "USR-999", "payload": "{\"tier\": \"EXPIRED\", \"amount\": 50.00}", "expectedStatus": "REJECTED_UNAUTHORIZED" },
  { "inputUserId": null, "payload": "{\"tier\": \"BASIC\", \"amount\": 10.00}", "expectedStatus": "REJECTED_NULL_ID" }
]

Your automated evaluation test reads this dataset, passes every payload through the AI-generated service, and asserts the outcomes. When you refactor prompts or upgrade your model version, running this dataset guarantees zero regressions.

Watch Out for Rejection Bias: When building your golden dataset, it is easy to overload it with edge cases that must be rejected (nulls, empty strings, malformed syntax). If your dataset is heavily skewed toward rejections, you risk driving the AI validator toward the worst possible convergence: a service that rejects everything. A system that says "no" to every input has zero false positives on rejections, but it is completely useless. Ensure that the cases which must be accepted carry just as much weight and diversity as the ones that fail.

3. Beyond Correctness: The Production Audit Checklist

Once the code passes your functional tests, you must conduct a targeted code review focusing on dimensions that automated tests often miss:

Resource Management & Memory: Did the model properly handle connection closures, stream disposal, or memory lifecycle hooks? Look out for unclosed handles or collections accumulating state indefinitely.
Concurrency & Thread Safety: LLMs frequently default to non-thread-safe data structures in multi-threaded contexts. Ensure safe alternatives or proper synchronization mechanisms are in place.
Exception Hygiene: Does the code swallow broad exceptions or suppress stack traces? Verify that errors throw meaningful, domain-specific exceptions with appropriate logging context.

4. Treat Prompts as Architecture, Not Chat History

If your team relies heavily on AI workflows, automated prompt generation, or contextual system instructions, stop treating prompts like casual chat messages.

Version Control: Store your prompt templates in dedicated version-controlled directories rather than hardcoding them inline.
Consistency Metrics: Measure prompt reliability across multiple iterations before deploying changes that affect core business pipelines.

Conclusion

The mark of a senior engineer in the age of AI isn't how fast they can copy-paste generated code; it's how rigorously they can evaluate it.

Treat AI-generated code with the exact same professional skepticism you would apply to code submitted by an external contractor. Write your tests first, audit for real-world production constraints, and remember: The best developers using AI are not the ones who trust it blindly, but the ones who verify it with absolute precision.

Use SOLID Principles To Become Better Automation Tester

gusdmoreira — Thu, 24 Jun 2021 20:51:47 +0000

Are you using most recommended and fundamental SOLID principles in your automation?

Why You Need To Use SOLID Principles?

Like any development work, automation also needs to be planned, designed, developed, maintained and extended over time. Your automation code or automation framework is an application which is testing your actual application. That’s why SOLID principles are highly recommend for your automation to make it easy to understand, use, maintain and extend over time. When SOLID principles are not applied, your automation might become fragile, hard to maintain and hard to extend very soon.

Even if you are using different programming language like C#, Python or automation tool like HP UFT/QTP for automation, you should definitely consider applying SOLID principles to make your automation tester life easy. Please note that we will be discussing about applying SOLID principles to web automation using Selenium & Java. But, once you learn the concepts, you can easily apply them to any object-oriented development.

In this post, we will be exploring what SOLID principles stands for. We will be having one post for each principle with real life examples from web automation using Selenium & Java.

What Are The SOLID Principles?

SOLID principles are the first five object-oriented design principles by Robert C. Martin, popularly known as Uncle Bob.

SOLID is an acronym where:

S stands for Single Responsibility Principle (SRP)
O stands for Open Closed Principle (OCP)
L stands for Liskov Substitution Principle (LSP)
I stands for Interface Segregation Principle (ISP)
D stands for Dependency Inversion Principle (DIP)

Let’s explore and understand each individual principle. Also, let’s find out how these SOLID principles make you better automation tester.

Single Responsibility Principle (SRP): A class should have one, and only one, reason to change

In other words to achieve this, a class should only have a single responsibility and it should do that very well. Every class in your automation should only have a single responsibility and that all of its methods should be aligned with that responsibility.

Let’s try to understand this principle by looking at an example from our daily life. When you are driving a car/bike, you want to fully concentrate on the single responsibility – driving. You don’t want to do or concentrate on other tasks like talking on a phone, eating.

Automation Examples:

Page Object Framework implements SRP very well. We are going to have one class responsible for only one web page in the application. We shouldn’t have a very big class with many responsibilities like test methods, UI action methods, excel read/write methods.
We can have very specific helper classes like ExcelHelper, DatabaseHelper to work with a excel file or database to implement Data Driven Framework.
Selenium API has browser specific driver classes like FirefoxDriver, ChromeDriver, InternetExplorerDriver. FirefoxDriver has single responsibility to drive the Firefox browser and it drives the Firefox browser very well.

We should also have methods in classes that are very specific like loginAs(username, password) to login with given username and password, testSuccessfulLogin(username,password) which tests only one thing that login should be successful with valid username/password.

Open Closed Principle (OCP): Software entities (classes, modules, functions, etc.) should be open for extension, but closed for modification

You should be able to easily add additional functionality for a class without changing its code. OCP says that a class should be open for extension and closed for modification. The “closed” part of the rule states that once a class has been developed and tested, the class code shouldn’t change except for any bug fixes. The “open” part of the rule states that you should be able to extend existing code in order to introduce new functionality. We are trying to add new functionality without modifying the existing code/functionality and by adding new classes/code as required. This is very important to minimise the impact of changes and errors from existing code.

Let’s try to understand this principle by looking at an example from our daily life. Let’s say you live in a 2 bedroom house and you are looking for a 3 bedroom house due to reasons like growing children. If you have free/unused space available, it’s very easy to extend the house by building another bedroom. Also with this, you are minimise the impact of changes on the existing 2 bedroom house.

Automation Example:

Let’s say that you are automating an online store application. You have Customer class to represent store customers and respective related customer actions. Now, your company has introduced VIP customer concept to reward loyal customers with discounts and free delivery. To implement VIP customer behaviour in your automation, OCP suggests that keep the Customer class same without modifying it and create a new VipCustomer class by inheriting from Customer class. Now in the VipCustomer class extend the behaviour as required.

Liskov Substitution Principle (LSP): Subtypes must be substitutable for their base types

When you pass subtype for a base type argument or when you assign/instantiate base type with subtype, the program/code should work properly without changing its behaviour and shouldn’t break. This principle was introduced by and named after Barbara Liskov.

Let’s try to understand this principle by looking at an example from our daily life. Let’s say that you have a wall clock at home or you have a wrist watch. They both need batteries to work. If you buy batteries as per the specifications from any brand like Panasonic, Sony, Duracell, you expect the wall clock or wrist watch to work properly without any issues when powered by those batteries.

Automation Example:

Considering the above example of automating an online store application with Customer and VIP Customer categories, let’s say there is a method calculateDeliveryCharge(Customer customer, OrderInfo orderInfo) which calculates delivery charge when customer and order information is passed. When we pass Customer object to calculateDeliveryCharge() method with order information, it should return delivery charge. Note that we are providing free delivery to VIP Customers. So, when we pass VipCustomer object for customer argument to calculateDeliveryCharge() method with order information, the program/code should work properly without changing its behaviour and shouldn’t throw any exceptions.

Interface Segregation Principle (ISP): Make fine grained interfaces that are client specific

It’s good to have small role specific interfaces rather than one big general interface. ISP states that clients should never be forced to implement interfaces that they don’t use or clients should never be forced to depend on methods that they don’t use. When a class depends upon another class, the number of members visible from the another class to the dependent class should be minimised. When you apply the ISP, classes implement multiple smaller role specific interfaces and dependent classes depend on required role specific interfaces for the given task.

Let’s try to understand this principle by looking at an example from our daily life. When you are travelling in a train and when ticket inspector wants to check your ticket, you will be showing only your ticket and not all your luggage. Similarly, ticket inspector wants to check your ticket only and not any other belongings of you. We should reveal/expose only what’s required for the given task.

Automation Example:

Let’s again take the example of automating an online store application. You have to test that an email is received by the customer after successful purchase. Let’s say that you have an EmailHelper class with a method isEmailReceived().

Your Customer class might have so many fields and methods like firstName, lastName, emailAddress, phoneNumber, deliveryAddress, billingAddress, getPurchaseHisotry(), getOrdersInProgress(). But isEmailReceived() method only needs to know firstName, lastName, emailAddress fields and doesn’t need to now all the other fields/methods of Customer class.

Consider implementation like isEmailReceived(Customer customer, string subject, string body): We are exposing whole Customer class in isEmailReceived() method and then one can easily access/change details like phoneNumber, deliveryAddress, billingAddress in isEmailReceived() method. We want to minimize the risk to Customer object by passing only details that are required to the clients/methods. We can achieve this by below implementation by using an Emailable interface which represents a contact that can be emailed.

interface Emailable {
    String getFirstName();
    String getLastName();
    String getEmailAddress();
}

class Customer implements Emailable {
    String firstName, lastName, emailAdddress;
    String phoneNumber, deliveryAddress, billingAddress;

    public String getFirstName() {
        return firstName;
    }

    public String getLastName() {
        return lastName;
    }

    public String getEmailAddress() {
        return emailAdddress;
    }
}

class EmailHelper {
    public boolean isEmailReceived(Emailable recipient, String subject, String body) {
        // Logic to check whether recipient recieved email or not

        // You can use recipient.getFirstName(), recipient.getLastName(), recipient.getEmailAddress() to get recipient details

        // When you pass Customer object to this method, only Emailable interface methos are available here and no other details are exposed from Customer Object

        // Customer object is safe from any changes

        return false;
    }
}

Now, we can further reuse Emailable interface for other types like StoreManager, Supplier to whom your online store sends emails. With this approach, we can use isEmailReceived() method with any implementation that implements Emailable interface and respective input object for recipient argument is safe from any changes/actions. I hope now you have understood the power of ISP.

Other Automation Example:

Selenium API has good examples of ISP. Selenium API has a number of very fine grained, role based client specific interfaces like WebDriver, WebElement, Alert. We should favour role based interfaces instead of generic interfaces.

Dependency Inversion Principle (DIP): Depend on abstractions, not on concretions

DIP states that Software entities (classes, modules, functions, etc.) should depend on abstractions (like interfaces) and not on concretions (like concrete class types). DIP promotes code to an interface approach. For example, in Selenium automation code, we code to a WebDriver interface variable “driver” whenever we want to work with web browser and the same code works for any browser type like FirefoxDriver, ChromeDriver, InternetExploerDriver which implements the WebDriver interface.

DIP mainly suggests below 2 rules:

High level modules should not depend upon low level modules. Both should depend upon abstractions.
Abstractions should not depend upon details. Details should depend upon abstractions.

Let’s try to understand this principle by looking at an example from our daily life. When you go to a cash machine/ATM, the cash machine/ATM expects a valid debit/credit card. The machine has a dependency on valid card abstraction and not on specific concrete type cards like only Visa, only Maestro or only issued by specific bank. The machine works for any valid card type implementation and we are providing the card to the machine from outside which provides so much flexibility and easiness to use the machine.

Automation Example:

Look at below page object class for a Login Page.

public class LoginPage {
    private WebDriver driver;

    public LoginPage(WebDriver driver) {
        this.driver = driver;
    }

    public void loginAs(String username, String password) {
        driver.findElement(By.id("username")).sendKeys(username);
        driver.findElement(By.id("password")).sendKeys(password);
        driver.findElement(By.name("login")).click();
    }
}

We can observe following with respect to DIP from above LoginPage class:

It is depending on the abstraction and all the code is written against WebDriver interface so that the code can work with any implementation like FirefoxDriver, ChromeDriver, InternetExploerDriver
The implementation for the WebDriver interface should be passed by the client through constructor LoginPage(WebDriver driver) when creating the LoginPage object.
So the code works for any implementation and we can pass the implementation when creating the object. This gives us lot of flexibility and it will be easy to maintain.

Get Ready To Use/Apply SOLID Principles In Your Automation…

I hope you have got good understanding of SOLID principles by now. If it seems complicated, don’t worry and things will be very clear in the next posts. Let’s look at each principle individually in the next posts to find out how these principles makes us better automation testers and makes our automation easy to understand, use, maintain and extend.