Most of us have a task we do so often that we stopped noticing it. Copying a failing test name into a ticket. Reformatting the same log output. Pasting the same checklist into every pull request.
Here is a hypothetical to make it concrete. Imagine a QA engineer who, after every failed nightly run, opens the CI log, finds the failing tests, checks whether each one failed last night too, and posts a summary in a team channel. It takes fifteen minutes. It happens every morning. Nobody questions it because it's "just part of the job."
That's the moment worth a second look.
The signal: repetition plus a stable pattern
Not every repeated task deserves automation. Three questions help me decide:
- Is the input predictable? Does it come from the same place in roughly the same shape?
- Are the steps mostly the same each time? Or does each run need real judgment?
- Is the cost of a mistake low or easy to catch? A wrong summary is cheap. A wrong production change is not.
If the answer is yes to all three, the task is a candidate. If the input is messy or the decision is high-stakes, keep a human in charge and use tooling only to assist.
Step one: write the task down before touching code
Before building anything, describe the task in plain steps. For the nightly-failure example:
- Fetch the latest test report.
- List failed tests.
- Compare with the previous run.
- Label each as "new failure" or "still failing."
- Post a summary.
Writing it out often reveals that most of the work is mechanical. Only one step, working out why something failed, needs a person.
Step two: automate the boring part first
A test report here means a structured file, commonly JUnit XML, that most test frameworks can produce. Because it's structured, a small script can read it reliably. No AI needed.
Here is an illustrative Python sketch:
import xml.etree.ElementTree as ET
def failed_tests(path):
tree = ET.parse(path)
failures = set()
for case in tree.iter("testcase"):
if case.find("failure") is not None or case.find("error") is not None:
failures.add(f'{case.get("classname")}.{case.get("name")}')
return failures
today = failed_tests("reports/today.xml")
yesterday = failed_tests("reports/yesterday.xml")
new_failures = sorted(today - yesterday)
still_failing = sorted(today & yesterday)
print("New failures:", *new_failures, sep="\n ")
print("Still failing:", *still_failing, sep="\n ")
This is a sketch, not a drop-in tool. File paths, report layout, and how you store yesterday's report will differ in your setup. But the shape is the point: set comparison replaces a manual scan.
Step three: ask where AI actually helps
Once the mechanical work is scripted, look at what's left. In our example, it's the "why did this fail?" question.
This is where a language model (an AI system that generates text from a prompt) can assist. You might pass it the error message and stack trace and ask for a short, plain-language hypothesis.
A few practical rules:
- Treat the output as a suggestion. Models can sound confident and be wrong.
- Send only what's needed. Strip secrets, tokens, and customer data from logs before they leave your environment. Check your organization's policy first.
- Keep the prompt narrow. "Summarize this error in two sentences and list likely causes" beats "analyze this."
- Label AI-generated text in the summary so readers know it's a hypothesis.
An example prompt:
You are helping triage a failed automated test.
Given the error and stack trace below, write:
1. A two-sentence summary of what failed.
2. Up to three possible causes, most likely first.
Do not claim certainty. If the information is insufficient, say so.
Error:
{error_text}
The last instruction matters. Asking the model to admit uncertainty reduces the chance of a tidy but misleading answer.
Step four: keep a quality check on the automation
Automation is also code, so it needs the same care as anything else.
- Test the parser with a known report, including an empty one and a malformed one.
- Make failures loud. A script that silently posts nothing is worse than no script.
- Review a sample of AI-written summaries periodically against what actually went wrong.
This is the quality engineering mindset applied to your own tooling: if it saves time but can't be trusted, it hasn't saved anything.
When to leave it alone
Sometimes the second look says "don't." If a task happens twice a year, the script costs more than it saves. If the task builds understanding you need, such as a new engineer reading logs to learn the system, automating it removes the learning. Repetition alone is not a reason.
A simple habit to start with
Keep a running note of tasks you repeat. After a few weeks, scan it for ones that are frequent, predictable, and low-risk. Pick the smallest one and automate just the mechanical part.
Which repeated task on your team have you stopped questioning, and what would a second look reveal?
Top comments (0)