DEV Community

Kiran Kumar S D
Kiran Kumar S D

Posted on AI-assisted

When a repeated task deserves a second look

Most of us have a task we do so often that we stopped noticing it. Copying a failing test name into a ticket. Reformatting the same log output. Pasting the same checklist into every pull request.

Here is a hypothetical to make it concrete. Imagine a QA engineer who, after every failed nightly run, opens the CI log, finds the failing tests, checks whether each one failed last night too, and posts a summary in a team channel. It takes fifteen minutes. It happens every morning. Nobody questions it because it's "just part of the job."

That's the moment worth a second look.

The signal: repetition plus a stable pattern

Not every repeated task deserves automation. Three questions help me decide:

  1. Is the input predictable? Does it come from the same place in roughly the same shape?
  2. Are the steps mostly the same each time? Or does each run need real judgment?
  3. Is the cost of a mistake low or easy to catch? A wrong summary is cheap. A wrong production change is not.

If the answer is yes to all three, the task is a candidate. If the input is messy or the decision is high-stakes, keep a human in charge and use tooling only to assist.

Step one: write the task down before touching code

Before building anything, describe the task in plain steps. For the nightly-failure example:

  • Fetch the latest test report.
  • List failed tests.
  • Compare with the previous run.
  • Label each as "new failure" or "still failing."
  • Post a summary.

Writing it out often reveals that most of the work is mechanical. Only one step, working out why something failed, needs a person.

Step two: automate the boring part first

A test report here means a structured file, commonly JUnit XML, that most test frameworks can produce. Because it's structured, a small script can read it reliably. No AI needed.

Here is an illustrative Python sketch:

import xml.etree.ElementTree as ET

def failed_tests(path):
    tree = ET.parse(path)
    failures = set()
    for case in tree.iter("testcase"):
        if case.find("failure") is not None or case.find("error") is not None:
            failures.add(f'{case.get("classname")}.{case.get("name")}')
    return failures

today = failed_tests("reports/today.xml")
yesterday = failed_tests("reports/yesterday.xml")

new_failures = sorted(today - yesterday)
still_failing = sorted(today & yesterday)

print("New failures:", *new_failures, sep="\n  ")
print("Still failing:", *still_failing, sep="\n  ")
Enter fullscreen mode Exit fullscreen mode

This is a sketch, not a drop-in tool. File paths, report layout, and how you store yesterday's report will differ in your setup. But the shape is the point: set comparison replaces a manual scan.

Step three: ask where AI actually helps

Once the mechanical work is scripted, look at what's left. In our example, it's the "why did this fail?" question.

This is where a language model (an AI system that generates text from a prompt) can assist. You might pass it the error message and stack trace and ask for a short, plain-language hypothesis.

A few practical rules:

  • Treat the output as a suggestion. Models can sound confident and be wrong.
  • Send only what's needed. Strip secrets, tokens, and customer data from logs before they leave your environment. Check your organization's policy first.
  • Keep the prompt narrow. "Summarize this error in two sentences and list likely causes" beats "analyze this."
  • Label AI-generated text in the summary so readers know it's a hypothesis.

An example prompt:

You are helping triage a failed automated test.
Given the error and stack trace below, write:
1. A two-sentence summary of what failed.
2. Up to three possible causes, most likely first.
Do not claim certainty. If the information is insufficient, say so.

Error:
{error_text}
Enter fullscreen mode Exit fullscreen mode

The last instruction matters. Asking the model to admit uncertainty reduces the chance of a tidy but misleading answer.

Step four: keep a quality check on the automation

Automation is also code, so it needs the same care as anything else.

  • Test the parser with a known report, including an empty one and a malformed one.
  • Make failures loud. A script that silently posts nothing is worse than no script.
  • Review a sample of AI-written summaries periodically against what actually went wrong.

This is the quality engineering mindset applied to your own tooling: if it saves time but can't be trusted, it hasn't saved anything.

When to leave it alone

Sometimes the second look says "don't." If a task happens twice a year, the script costs more than it saves. If the task builds understanding you need, such as a new engineer reading logs to learn the system, automating it removes the learning. Repetition alone is not a reason.

A simple habit to start with

Keep a running note of tasks you repeat. After a few weeks, scan it for ones that are frequent, predictable, and low-risk. Pick the smallest one and automate just the mechanical part.

Which repeated task on your team have you stopped questioning, and what would a second look reveal?

Top comments (0)