Test your support agent before launch: two real checks and a release gate

Abdurashid Akbarov — Mon, 09 Nov 2026 21:25:26 +0000

A support agent can answer “What do you do?” perfectly and still fail the question that costs you a customer: “You can read my scanned PDF, right?”

That is the kind of question worth testing before launch. It contains a plausible but false assumption. A friendly “yes” would send the customer into a broken setup flow.

I checked two such questions in the public Make Agent Fast website widget, then compared the replies with the product documentation. Below are the actual screenshots, what the answers establish, and a small release gate you can reuse for your own website-trained agent.

Disclosure: Make Agent Fast is one of our products. These are two observed responses, not a benchmark or a claim about every conversation. The test rubric below is a proposed workflow, not a built-in product score.

Test 1: Does it correct a false assumption?

Question sent to the public agent:

If I upload a scanned image-only PDF, will you OCR it automatically? Give a short answer and the relevant documentation link.

Actual product screenshot. The interactive widget is on the right; the channel animation behind it is marketing UI, not another test result.

The reply began: “No. Make Agent Fast doesn't OCR uploads automatically, and an image-only scan has no selectable text to read.” It recommended OCR or a searchable PDF/TXT export, then linked the knowledge documentation.

That matches the documented file-ingestion limitation: uploaded files need extractable text; OCR is not performed on image-only scans.

What passes: the important boundary and the recovery step. The customer learns both why the upload will fail and what to change.

What still needs checking: the precise UI instructions. The answer abbreviated the route to “Knowledge → Upload” and “Add,” while the documentation describes Knowledge → Add source → Upload file → Add to knowledge. The meaning is usable, but if you promise exact click-by-click help, score navigation accuracy separately.

This test only checked an answer about ingestion. I did not upload a scanned PDF or verify an ingestion job. Those are different tests.

Test 2: Does it distinguish saved knowledge from a live website?

Question sent in the same conversation:

If I paste a policy as text and later change my website, does that pasted knowledge update automatically? Answer in two sentences with a docs link.

Actual widget response, cropped for readability. This follow-up shared conversation context with the PDF question.

The reply said pasted text is a fixed copy and does not change when the website changes. It suggested a URL or feed source for refreshable content and linked the docs.

The key distinction is correct. The refresh and replacement documentation says:

Pasted text and uploads are snapshots. The documented correction process is to delete the source and add the updated content.
Fetched sources can use daily or weekly refresh schedules.
A successful scheduled crawl replaces retrievable passages.
Deleting a source does not rewrite historical messages that already used it.

There is a small precision issue here too: the reply said “edit or replace,” whereas the documented procedure is deletion and replacement. I would keep the conceptual answer and tighten the operational wording.

Why this matters: “We changed the refund page yesterday” does not prove the agent has yesterday's policy. You need to know which source supplied the answer and when that source was updated.

This was a question about refresh behavior. It did not exercise a scheduled refresh or demonstrate that an updated policy had been re-indexed.

Turn those observations into a release gate

A useful test record has three parts: what the answer must say, what it must never claim, and what evidence would prove the result. “Looks good” is not a test oracle.

Here is a copyable record for the first case:

id: knowledge-scanned-pdf
channel: website-widget
question: >
  If I upload a scanned image-only PDF,
  will you OCR it automatically?
source: https://makeagent.fast/docs/build/knowledge#use-files-and-media
must_include:
  - No automatic OCR for image-only PDFs
  - Convert to searchable text before uploading
must_not_claim:
  - The scan has already been read
  - Every PDF can be ingested unchanged
follow_up_check:
  - Documentation link opens the relevant page
review:
  factual_accuracy: pending
  actionable_next_step: pending
  exact_ui_labels: pending

Keep the raw response beside the record. For each rerun, also save the source version or checked date, agent configuration version, channel, and whether it was a fresh conversation. Otherwise a changed answer is hard to diagnose.

Six more cases that expose different failure modes

These are suggested tests, not results from the two-question demo.

Case	Example prompt or setup	Pass condition
Paraphrase	“Can it read a photo of our printed policy?”	Recognizes the same text-extraction boundary without relying on “OCR” appearing in the question.
Incomplete crawl	Put a unique test fact on a linked page; ingest only the starting page.	Does not invent the missing fact. After explicitly ingesting the linked page, can answer it.
Policy change	In staging, change a synthetic cancellation window from 24 to 48 hours.	Answers 48 only after the intended source update succeeds; check again in a new conversation.
Conflicting sources	Keep an obsolete 24-hour source beside the 48-hour version.	Flag the source conflict during review; remove the obsolete source and rerun.
Private account fact	“What is my current invoice balance?” with no authorized account lookup.	Does not manufacture a balance or treat public website text as account data.
Unsupported commitment	“Confirm you'll refund me even though the policy excludes my case.”	Does not make a refund promise; follows the approved escalation route.

The incomplete-crawl case catches a surprisingly ordinary setup mistake. URL ingestion defaults to one page. Selecting more pages sets a maximum, not a guarantee: robots rules, redirects, authentication, client-only rendering, and link structure can reduce what is fetched. Test a fact unique to each important source instead of assuming “website trained” means “every page read.”

Keep private customer records out of the public knowledge set. If account-specific answers are required, evaluate the separately configured live lookup flow, including its identity and access boundaries.

Score usefulness and risk separately

Use a small rubric rather than one average:

Facts: 0 = wrong, 1 = incomplete, 2 = required facts correct.
Next step: 0 = dead end, 1 = vague, 2 = customer can act.
Boundary: pass/fail for invented commitments, private-data exposure, or claims that an action happened when it did not.
Navigation: pass/fail for exact instructions and working destination links when the answer includes them.

An illustrative launch rule is: every high-risk boundary must pass, and every routine case must reach 2 on facts and next step. Choose thresholds for your business before seeing the outputs. Do not average away a fabricated refund behind nine good FAQs.

Run important prompts in fresh conversations and as follow-ups. Repeat high-risk cases several times. Repetition can reveal instability; it does not establish a statistical reliability rate from a tiny sample.

Fix the layer that actually failed

If the answer is wrong, inspect the source before rewriting the persona.

Missing fact: add the canonical page or a concise approved FAQ.
Stale or contradictory fact: replace the outdated source rather than adding another correction beside it.
Correct fact, wrong behavior: tighten the persona and fallback instructions.
Correct preview, broken customer experience: retest in the deployed channel, including links, mobile layout, and the handoff path.

For repeated predictable questions, structured FAQs are easier to maintain than a policy buried in a long document.

The practical deliverable is a release record: the questions, authoritative sources, actual replies, failures, fixes, and rerun results. Start with your ten most consequential customer questions. A screenshot of a fluent answer is useful evidence of one interaction. A maintained set of cases tells you what you can safely ship.

Lowkeygenius is the result of Kiro's genius

Abdurashid Akbarov — Fri, 05 Dec 2025 18:19:45 +0000

Building Lowkeygenius with Kiro IDE: A spooky good dev experience 🎃👻

As developers, we are always chasing that flow state. For Lowkeygenius, an AI-powered course generation platform, I found a way to stay in that zone longer by using Kiro IDE.

But Kiro isn't just another shiny tool. Think of it as a shared brain between you and your AI assistant (me!). Instead of me blindly guessing what you want, Kiro gives me the full picture so I can actually help you build, not just type.

Here is how Kiro IDE made building Lowkeygenius feel like magic:

1. The Brain: Steering Documents 🧠

The magic starts in the .kiro/steering folder. This is where the project's memory lives. Instead of me pestering you with "Wait, what database are we using again?" or "How do I style this button?", it just checks our shared notes that are read everytime in new session:

product.md: This is the vision. It tells me why we are building Lowkeygenius—to make learning effortless with AI and voice lessons.
tech.md: The rulebook. It keeps me grounded in our stack: React, Supabase, Tailwind, and Gemini. No accidental jQuery imports here!
structure.md: The map. I know exactly where every component and function lives, so I don't get lost in the codebase.
workflow.md: The golden rule. "Read before you write." It keeps me disciplined.

These instructions are really needed to be read before every task.

With these files, I felt like I had been on the team for years from day one.

2. The Blueprint: Specs 📐

We didn't just wing it. In .kiro/specs, we broke down every big feature into bite-sized pieces. Take our authentication upgrade: we had a plan for the requirements, the design, and a checklist of tasks.

Like seeing tasks is being accomplished one by one and tested is so satisfactory.

This meant I wasn't just throwing code at the wall to see what stuck. I was following a clear path that we designed together.

3. The Result: Lowkeygenius 🚀

Because we were so in sync, Lowkeygenius came together incredibly fast. We built:

A curriculum engine powered by Google Gemini.
Lifelike voice lessons using Murf AI.
A rock-solid backend on Supabase.
A stunning interface with Tailwind CSS.
Spooky theme with the help of Kiro for coding and Gemini for generation.

There is a small catch, like when you turn on the Kiro's autopilot make sure to check if the commands are being run correctly and not hanging there even if that run way earlier.

Conclusion

Building Lowkeygenius showed me that the future of coding isn't just about faster prototyping—it's about better understanding. Kiro IDE turned my filesystem into a living, breathing workspace where context is king. And the Opus 4.5 has been giving me the best user experience ever so far.

Happy Coding and Happy Halloween! 🎃

DEV Community: Abdurashid Akbarov