DEV Community: Andrey Vishin

The bugs that look like success: quality checks for AI-generated learning content

Andrey Vishin — Thu, 22 Oct 2026 15:37:44 +0000

The card that passed every check

During the final end-to-end test before LAFwords launched, I tried a deliberately broken input. I typed asdfgh as a word to learn, and when the app asked whether I meant something else, I chose to keep my original word.

The app built a card anyway. It had ten example sentences, an AI image, pronunciation audio and a success dialog, and it went into my study queue. Here are some of the sentences, with the word the card was teaching in bold:

"I'm sorry, but asdfgh isn't a recognized English word, so I can't create valid usage samples for it."
"Please provide a real English word so I can generate varied usage samples."
"Tell me a real target word and I will produce ten diverse sentences."

Every sentence was a piece of the model's apology. And every automated check had passed. All the words were real dictionary words. The entered word appeared in a sentence. There were enough sentences.

That bug is the best summary I have of what testing LLM output is like. The dangerous failures don't look like failures. They look like success.

This post is about how I check the content LAFwords generates: what broke, what each break taught me, and what my checks still don't catch. My background is in software QA, with hands-on work in prompt engineering and in evaluating non-deterministic LLM output. This has been the most interesting testing problem I've worked on.

What gets generated, and why errors cost more here

LAFwords is a vocabulary app for intermediate and advanced learners. You enter a word you met, in English or Spanish, and the app builds a card: ten sentences that show the word in different situations, each with a short meaning, a context label and an image, plus native pronunciation. There are no translations. Ten varied contexts give you a feel for the word's range, so you can use it in situations the ten sentences never showed.

That design allows no compromise on quality. A learner can't spot the errors because they don't know the word yet. If a sentence uses the word in the wrong sense, or invents a form that doesn't exist, the learner memorizes the mistake. The generated content is the curriculum.

1. The model can't check its own inventions

It gets worse: language models are prone to inventing word forms. They apply patterns like -ity and -ness that work for thousands of words, but real vocabulary doesn't follow them everywhere—borrowed words and one-off formations break the rules.

Early on, the model produced holisticity (for holistic) and rigorousnesses (for rigorous). Neither is an English word. Both look plausible, which is the problem.

The obvious idea is: let's ask the model whether a word is real! It doesn't work reliably. The reason is simple: we judge the invented word using the same model knowledge that generated it. A checker that shares the generator's blind spot passes exactly the errors you most need to catch.

So the first lesson I learned: don't let the model check itself. The fabrication check doesn't use an LLM at all. Every word in every sentence is checked against real spelling dictionaries, the same kind a word processor uses. The dictionary brings ground truth from outside the model. If a sentence contains a word the dictionary doesn't know, it's rejected and regenerated.

A side lesson came from the re-test. The model used rigorousness, and my first reaction was that it was another invention. It isn't. It's in the dictionary. My instinct as a reviewer was no better than the model's, and only the external source could decide.

2. The quality rule that caused the defect

Why was the model inventing words in the first place? Because I asked it to.

The early prompt required several different forms of the word across the ten sentences, so the learner would see decide, decided, deciding, decides. It sounded like good teaching. But holistic has only two real forms: holistic and holistically. When a requirement can't be met with real words, the model doesn't refuse. It fills the gap with plausible inventions.

Removing the requirement fixed it. A re-test of holistic produced the two real forms and nothing else. Different forms are still welcome when they appear naturally, but they are never forced.

So, the next lesson: keep your eye on edge cases where the input makes the constraint impossible to follow. Ask what the model will do. Unfortunately, it usually sticks to the letter of the rule and pays for it in correctness.

3. Reasonable gates, garbage output

Back to asdfgh. Each check in the pipeline was sensible on its own. Are all the words real? Yes, an apology is made of real words. Does the entered word appear? Yes, once, inside the apology. Are there enough sentences? Yes.

What I missed was checking the relationship between each sentence and the target. Every sentence highlights the word it teaches. In the broken card, the highlighted words were please, tell, choose: real words, just not the word the card was about.

The fixes were specific. Each sentence's highlighted word must now match the entered word or a form of it, and anything else is rejected. Known refusal phrasings are detected and rejected too. And the exact sentences from that card became a regression test, so this failure can't quietly come back.

The lesson I took here: a list of property checks is not a specification. Each gate checked that something looked normal. None checked that the output did its job.

4. When the check measures a proxy

In September, I noticed a card for triage in production with ten medical sentences: the ER, a field clinic, a disaster drill, a triage nurse. Every sentence was correct. But triage has a second sense that an office worker probably meets more often: prioritizing anything scarce, like a bug backlog, an inbox, or a budget. That sense was missing entirely.

Ten sentences in one sense defeat the point of the card. They teach one corner of the word ten times.

The worse news was in my review checklist. It checked variety by counting distinct context labels, and the triage card had ten different labels: "ER", "disaster response", "drill" and so on. By my own checklist, the card passed. The checklist measured a proxy, labels, instead of the thing itself: meanings.

The cause turned out to be a model setting. To save cost during bulk generation, the model ran with low reasoning effort, and at that level it follows its first association and stays there. The original card had zero non-medical sentences out of ten, and a regeneration had two. After raising the setting one step, the next generation gave five: IT support, a newsroom, email, a budget, project planning.

Two lessons. When a check uses a proxy, test it against a known bad case before trusting it. And a setting chosen for cost is a quality decision, whether or not anyone treats it as one.

5. Strict gates fail silently

The dictionary check is the strongest gate I have, and it has a cost I only found by looking for it.

During Spanish testing, the dictionary rejected four real words as inventions: afrutadas (fruity), carísimos (very expensive), desglasar (to deglaze) and risotto. Spanish builds forms like carísimo freely, and dictionaries don't list them all. Loanwords and specialist terms are often missing too.

Nobody noticed, because nothing visibly broke. The rejected sentences were regenerated, the card still had ten sentences, and it shipped. But the replacements were plainer. A strict gate quietly strips out exactly the rich, natural vocabulary an advanced learner wants.

English has the same problem. Our English dictionary doesn't know triaging, a perfectly regular form. And the check that catches refusal text once threw out "I cannot condone lying to my friends", a natural sentence for condone, because it sounded like the model saying no. In both cases the sentence was quietly replaced, and the card looked fine.

A gate's misses show up as bugs. Its false alarms show up as nothing at all.

My lesson here: watch what the validator removes, not only what it lets through.

6. Every new language reopens old failures

Spanish needed its own dictionaries and its own rules. Accents, for example, are part of the word: sí and si, tú and tu, papá and papa are different words. A human review of generated Spanish cards then found failure types that English testing never showed:

Another language leaking in. Italian congiuntivo instead of Spanish subjuntivo in a grammar field, and Chinese characters inside a Spanish context label.
The model's doubt written into the output. A part-of-speech field that read "verbo? (incorrecto)".
One generic meaning copied onto all ten sentences, instead of the meaning the word has in each sentence.
A neighboring word taught instead of the target. restar (to subtract) for resto (remainder).
Gender that changes the meaning. La radio is broadcasting; el radio is a radius. A wrong article silently changes the word.

Most of these get past the dictionary, because it checks the sentence, not the metadata fields around it. Inside the sentences, the words involved are real.

Italian and German come next. German uses gender to separate meanings even more often than Spanish does. I treat the list of past failures as the first draft of the test plan for each new language.

7. When the prompt and the validator disagree

In October, during a large import of new words for the app's word library, more than half of the cards came out short: six, seven, eight, or nine sentences instead of ten. There were no errors and no timeouts. I noticed it in the admin word list, where sixes and sevens kept showing up. The logs gave the scale: of 698 words generated in twelve hours, only 326 had all ten.

The cause was two rules that disagreed. The prompt allowed close relatives of the word: decide, decision, decisive. The validator, written after asdfgh, accepts only grammatical forms of the same word, so it rejected abhorrence on a card for abhorrent. The target-word check caused 907 of 1,045 rejections, and most of them were derived words the prompt had invited. Both rules were reasonable, and both sat in the same methodology document, a few lines apart. Nobody noticed that they contradicted each other.

Rejections alone would not have produced short cards, because a retry loop refills rejected slots. But in the batch import, one setting meant two things: the minimum number of sentences a card needs to ship, and the point where generation stopped trying. Once six sentences passed, the loop stopped. It took both bugs together to ship short cards, and every one of them passed as finished.

We chose one rule: a card shows the entered word and its grammatical forms, never derived words, which belong on their own cards. Generation now keeps going until there are ten sentences or the retry limit is reached. Replaying the old rejections through the new validator showed that 93 of them had been real forms of the word all along, lost to a gap in the tool that maps a word to its dictionary form. We paused the import, fixed the rules, and regenerated 833 cards before any images were made, so no image money was spent twice. Of the 2,599 imported words, 2,594 now have ten sentences.

The lesson: the generator's instructions and the checker's rules are one specification, so test them against each other, starting with every example in the prompt. Running decision for decide through the validator would have failed in a second. And watch the distribution of your output, not only pass and fail. A column full of sixes and sevens told me more than any single check.

What the checks still miss

Now the honest part. The lessons above apply to almost any LLM product. What follows doesn't. These are gaps specific to a vocabulary app, and that is exactly why they are still open: problems rooted in the domain don't give way to general rules. I list them less as lessons than as a sample of how varied quality problems get, even inside one narrow product:

Sense variety has no automated check. It depends on the prompt and the model setting. The triage fix made it better, but it's not guaranteed. This is the biggest open gap.
Sentences about the word instead of with it. For example, instead of using the word bass, the model defined it: "The term bass often refers to the lowest range in a musical ensemble." Every word is real, and the target is present, so the gates pass it. Only the prompt discourages it.
Words outside the word family. Mousetrap is highlighted as a form of mice. It's a real word, so the dictionary accepts it.
Grammar. A dictionary checks words, not how they combine. The same grammar mistake repeated across a card gets through.
Sensitive words. The classifier that flags vulgar or offensive words under-flags in practice.
Images that contradict the sentence, such as a person of a different gender than the sentence describes.

If you are new to evaluating non-deterministic LLM output, expect a list like this in your own domain. It will be different, and it won't be shorter.

Some of these problems can get a cheap mechanical check: a question mark in a grammar field is easy to catch. Others need a semantic judgment, and that is where an LLM judge may finally earn a place. Not to decide whether a word exists, but whether ten sentences really cover different meanings.

As long as these gaps remain, human review covers what the gates don't.

How I keep it honest

Three habits hold this together. None of them is clever. They just stop knowledge from leaking away between one fix and the next.

A failure gallery. Every real failure goes into one file: the actual output (verbatim, where I still have it), why it passed the checks, what caused it, and the rule it produced. Real cases only, no hypotheticals. Entries are never deleted when fixed, only marked fixed, because fixed failures come back.
One methodology document with a decision log. What makes a good card is written down in one place, with the date and the reason for every rule. Any change to the generation prompt or a validator must update that document in the same pull request. On its own, it didn't prevent the contradiction in section 7; that takes tests, not just a document.
Regression tests built from real output. The asdfgh sentences are a test fixture. A real failure makes a better test than one I would invent, because I would never have invented it.

Much of the day-to-day review is done with AI assistants, which makes the written rules more important, not less. An assistant reviewing cards follows the same checklist a human would, and the triage story shows what happens when the checklist is wrong.

If you're building something similar

Don't let the model grade its own facts. Bring ground truth from outside: dictionaries, schemas, a database, anything the model didn't write.
Read every prompt constraint as a test case. What does the model do when the input makes the constraint impossible?
Check that the output does its job, not only that it looks normal.
Test each check against a known bad case, especially when it measures a proxy.
Watch what your gates remove. False alarms cost quality that nobody sees.
Keep real failures, verbatim. They are the best test plan you will have for the next model, the next prompt or the next language.
Test the generator's instructions against the checker's rules. Start with every example in the prompt; one document is not enough.
Look at the distribution of your output, not only at pass and fail. A histogram of results can show a bug that no single check reports.

LAFwords is live at lafwords.com for learners of English and Spanish. If you work on LLM quality and see a gap in this approach, I'd like to hear about it.

Originally published on the LAFwords blog.

Using OTP sent via email in E2E UI tests.

Andrey Vishin — Tue, 24 Jan 2023 06:09:41 +0000

Background

When implementing end-to-end automated testing of an application with one-time password (OTP) authentication, we need to establish a way to test the entire flow of OTP delivery to the user.
Here I provide two classes that are ready to be used in this kind of test to receive OTP via email sent by the backend and a step-by-step guide on integrating into your framework.

Constraints

To implement this approach, the following conditions must be met:

The QA team manages a dedicated email account.
Emails delivering OTP have a known constant Subject text
A known constant phrase precedes OTP in the body of the email
The OTP length is fixed

Algorithm

Along execution, an autotest:

Creates and keeps an instance of the Email Provider service based on the configuration settings (Email Provider API credentials)
Gets and keeps a pointer to the last received email with a subject that is set for OTP emails (or retains null if no one was received before)
Triggers the OTP generation and delivery
Waits for email delivery to the Email Provider
Gets the email message
Parses OTP from the message
Uses OTP to verify login to the application under test

Steps 1-2 and 4-6 are implemented in the code provided.

EmailProviderHandler interface

To make the approach flexible, the EmailProviderHandler interface has been declared. The implementation must provide the following:

a service initiation to expect emails with specified subject
checking if a new email with the subject has already been received
getting the message from the received email

Interface

interface EmailProviderHandler {
    void init(String emailSubject);
    boolean isEmailReceived();
    String getMessage();
}

I have implemented the interface for Gmail (see below), but you can implement it for another provider you use.

EmailedOTPHandler class

To process emails with OTP, use the EmailedOTPHandler class.
An EmailedOTPHandler instance should be created for a specific combination of an email Subject, a passphrase followed by an OTP in the email body, and an OTP length. To give the handler a tool to access emails on a particular Email Provider, we should inject an instance of the implemented EmailProviderHandler.

Class variables list

    private final String emailSubject; // Subject pattern of the emails containing the OTP
    private final String otpKeyPhrase; // Phrase that precedes the OTP in the email body
    private final int otpLength; // Length of the OTP
    private final EmailProviderHandler emailProvider; // Gmail service

Before using an EmailedOTPHandler instance for getting OTP, you need to initiate it using the init() method.
Then you can trigger OTP generation and delivery by either mimic user login behavior through your application's UI or by querying the BE endpoint.
To get the OTP from an email, use the getOTPEmailSent() method. The method waits for a new email with Subject set and then tries to parse the OTP from it.
If there is no new message within the time period, NULL is returned.

The complete class code is shown below.

GmailHandler class

GmailHandler implements EmailProviderHandler to handle a Gmail service through API.
The methods described in the Google Gmail Java Quick Start guide are used to start the Gmail service and get credentials.

On the first Gmail API call, GmailHandler creates a credential file in the project to authenticate all future calls to the Gmail service (see detailed description below).

The complete class code is shown below.

Repository

Java (used in this article)
Python

Notice

To enable Java assert validation, use the JVM parameter -ea

IntelliJ IDEA

Before using the class, you must enable and configure the API for your Gmail account, as shown below.
GmailHandler extracts OTP from the email snippet. If your OTP in the email body is too far from the beginning and therefore not included in the snippet, use getPayload() instead of getSnippet().

How to set up the Gmail account API

Before moving on, you must activate and configure the API for the Gmail account you will be using to receive OTP emails. Using the Google Cloud Console follow the steps below.

Register a new project

Step by step

Click on CREATE PROJECT.

Then give your project a name.

Enable API

Step by step

Click on ENABLE APIS AND SERVICES.

Search for Gmail in the API Library.

Enable Gmail API

Create credentials for autotests to access your Gmail account

Step by step

Click on CREATE CREDENTIALS.

Choose for Gmail API a User data type.

Customize the OAuth Consent Screen - enter any name for the app and add your contact email address

Set the scope It make sense to choose the read only scope

Choose the Desktop app application type and give it a name

Your credentials have been created; you need to download the Client ID file in JSON format.

You can also customize your credentials at any time in the Credentials tab and then download the updated JSON file.

Register a trusted test user

Step by step

Navigate to the OAuth consent screen tab and click on ADD USERS

Add your any real Gmail account email address. You will need to act under this account later to verify access for the Client ID

How to add the Gmail account credentials to the project

After receiving the Client ID file in JSON format (as shown above), you must exchange it for the StoredCredential file the first time you call the Gmail API.

Add JSON Client ID file

Step by step

Put the file into src/main/resources/credentials.

Verify the autotest access to the account on Gmail

Step by step
Run your project first time. At the first call to the Gmail API, a browser will be opened by Google. Your should follow the Google dialog.

Choose the account you add on the Register a trusted test user step.

Click on Continue to verify the app

Click on Continue to grant the access

Check for the confirmation

Stop the first test project execution.

Check for the StoredCredential file

Step by step

The StoredCredential file should already be created automatically in src/main/resources/credentials during your first Gmail API call; if it is not, repeat this section again.

If you change the Gmail API configuration in the console in the future, you should delete the StoredCredential file and repeat these steps to add a new one.

Full code

EmailedOTPHandler

Code

package gmail;

import static java.lang.Thread.sleep;

interface EmailProviderHandler {
    void init(String emailSubject);
    boolean isEmailReceived();
    String getMessage();
}

public class EmailedOTPHandler {
    private final String emailSubject; // Subject pattern of the emails containing the OTP
    private final String otpKeyPhrase; // Phrase that precedes the OTP in the email body
    private final int otpLength; // Length of the OTP
    private final EmailProviderHandler emailProvider; // Gmail service

    public EmailedOTPHandler(String emailSubject, String otpKeyPhrase, int otpLength, EmailProviderHandler emailProvider) {
        this.emailSubject = emailSubject;
        this.otpKeyPhrase = otpKeyPhrase;
        this.otpLength = otpLength;
        this.emailProvider = emailProvider;
    }

    public void init() {
        emailProvider.init(emailSubject);
    }

    /**
     * Parse and return OTP from the email with id = this.emailID
     */
    private String parseOTP() {
        String mailText = emailProvider.getMessage();
        // Parse OTP
        int pos = mailText.indexOf(otpKeyPhrase);   // Find the OTP key phrase
        assert pos != -1 : "OTP key phrase not found in the email";
        pos = pos + otpKeyPhrase.length();         // Move to the OTP start position
        return mailText.substring(pos, pos + otpLength);
    }

    /**
     * Trying to get a new email, checking for a new message every 5 sec for 6 times.
     * If gotten a new message, return the OTP from it.
     * If there is no new message during the time period, return NULL
     */
    public String getOTP() {
        String otp = null;
        for (int attempt = 0; attempt < 6; attempt++) {
            if (emailProvider.isEmailReceived()) {
                otp = parseOTP();
                break;
            }
            try {
                sleep(5000);
            } catch (InterruptedException e) {
                e.printStackTrace();
            }
        }
        return otp;
    }
}

GmailHandler

Code

package gmail;

import com.google.api.client.auth.oauth2.Credential;
import com.google.api.client.extensions.java6.auth.oauth2.AuthorizationCodeInstalledApp;
import com.google.api.client.extensions.jetty.auth.oauth2.LocalServerReceiver;
import com.google.api.client.googleapis.auth.oauth2.GoogleAuthorizationCodeFlow;
import com.google.api.client.googleapis.auth.oauth2.GoogleClientSecrets;
import com.google.api.client.googleapis.javanet.GoogleNetHttpTransport;
import com.google.api.client.http.javanet.NetHttpTransport;
import com.google.api.client.json.JsonFactory;
import com.google.api.client.json.jackson2.JacksonFactory;
import com.google.api.client.util.store.FileDataStoreFactory;
import com.google.api.services.gmail.Gmail;
import com.google.api.services.gmail.GmailScopes;
import com.google.api.services.gmail.model.ListMessagesResponse;
import com.google.api.services.gmail.model.Message;

import java.io.*;
import java.nio.file.Files;
import java.security.GeneralSecurityException;
import java.util.*;

public class GmailHandler implements EmailProviderHandler{
    private static final String GMAIL_AUTHENTICATED_USER = "me";
    private final JsonFactory JSON_FACTORY = JacksonFactory.getDefaultInstance();
    private static final String APPLICATION_NAME = "Gmail handler";
    private static final String PATH_TOKEN_DIRECTORY = System.getProperty("user.dir") + "/src/main/resources/credentials";
    private static final String PATH_CREDENTIALS_FILE = PATH_TOKEN_DIRECTORY + "/gmail_credentials.json";
    private Gmail gmailService;
    private final List<String> SCOPES = Arrays.asList(GmailScopes.MAIL_GOOGLE_COM);
    private String emailSubject;
    private String emailID; // ID of the last email with the provided Subject


    public void init(String emailSubject) {
        this.emailSubject = emailSubject;
        startService();
        emailID = getEmailID();
    }

    /**
     * Start a new Gmail service
     */
    public void startService () {
        final NetHttpTransport HTTP_TRANSPORT;
        try {
            HTTP_TRANSPORT = GoogleNetHttpTransport.newTrustedTransport();
            gmailService =  new Gmail.Builder(HTTP_TRANSPORT, JSON_FACTORY, getCredentials(HTTP_TRANSPORT))
                    .setApplicationName(APPLICATION_NAME)
                    .build();
        } catch (IOException | GeneralSecurityException e) {
            throw new RuntimeException(e);
        }
    }

    /**
     * Creates an authorized Credential object.
     * This is the code from the Google Developer Console documentation.
     *
     * @param HTTP_TRANSPORT The network HTTP Transport.
     * @return An authorized Credential object.
     * @throws IOException If the credentials.json file cannot be found.
     */
    private Credential getCredentials(final NetHttpTransport HTTP_TRANSPORT) throws IOException {

        InputStream in = Files.newInputStream(new File(PATH_CREDENTIALS_FILE).toPath());
        if (in == null) {
            throw new FileNotFoundException("Resource not found: " + PATH_CREDENTIALS_FILE);
        }
        GoogleClientSecrets clientSecrets = GoogleClientSecrets.load(JSON_FACTORY, new InputStreamReader(in));

        // Build flow and trigger user authorization request.
        GoogleAuthorizationCodeFlow flow = new GoogleAuthorizationCodeFlow.Builder(
                HTTP_TRANSPORT, JSON_FACTORY, clientSecrets, SCOPES)
                .setDataStoreFactory(new FileDataStoreFactory(new File(PATH_TOKEN_DIRECTORY)))
                .setAccessType("offline")
                .build();
        LocalServerReceiver receiver = new LocalServerReceiver.Builder().setPort(8888).build();
        return new AuthorizationCodeInstalledApp(flow, receiver).authorize("user");
    }

    /**
     * Return ID of the last email with subject = emailSubject
     * @return - last email id
     */
    public String getEmailID() {
        List<Message> listOfMessages;
        try {
            ListMessagesResponse response = gmailService.users().messages()
                    .list(GMAIL_AUTHENTICATED_USER)
                    .setQ("subject:" + emailSubject)
                    .execute();
            listOfMessages = response.getMessages();
        } catch (IOException e) {
            throw new RuntimeException(e);
        }
        return (listOfMessages == null || listOfMessages.isEmpty()) ? null :  listOfMessages.get(0).getId();
    }

    /**
     * Return the email snippet text
     * @return - email snippet text as a String
     */
    public String getMessage() {
        Message message;
        try {
            message = gmailService.users().messages()
                    .get(GMAIL_AUTHENTICATED_USER, emailID)
                    .execute();
        } catch (IOException e) {
            throw new RuntimeException(e);
        }
        return message.getSnippet(); // Use getPayload() instead if you want to get the full email body
    }

    /**
     * Check if there is a new email with subject = emailSubject
     * @return
     */
    public boolean isEmailReceived() {
        String newEmailID = getEmailID();
        if (newEmailID == null) {
            return false;
        }
        if (newEmailID.equals(emailID)) {
            return false;
        }
        emailID = newEmailID;
        return true;
    }
}

Test example

Code

import gmail.EmailedOTPHandler;
import gmail.GmailHandler;

public class EmailedOTPTest {

    static String subject = "OTP test";
    static String keyPhrase = "Your OTP is: ";
    static int otpLength = 6;

    public static void main(String[] args) {

        GmailHandler gmail = new GmailHandler();
        EmailedOTPHandler otpHandler = new EmailedOTPHandler(subject, keyPhrase, otpLength, gmail);

        otpHandler.init();
        // -> Here trigger the OTP email sending
        String otp = otpHandler.getOTP();

        assert otp != null : "No new email with subject = '" + subject + "' was received during the time period";
        System.out.println("OTP: " + otp);
    }

}