DEV Community

Cover image for Survey Free-Text Classification: Schema, Confidence, and Re-Run Design
Lovanaut
Lovanaut

Posted on

Survey Free-Text Classification: Schema, Confidence, and Re-Run Design

#ai #data #automation #forms

Free-text survey and inquiry responses do not classify themselves, and a single "summarize this" prompt does not produce anything you can act on twice. What you need is a schema, a confidence threshold, and a re-run rule.

The Minimum Schema

Three fields, added to whatever the original submission already has:

category      -- one of a fixed, small set (pricing, onboarding, support, bug, other)
sentiment     -- positive | neutral | negative
confidence    -- 0.0-1.0, how sure the classification is
Enter fullscreen mode Exit fullscreen mode

Keep category closed. An open vocabulary just turns into a second free-text problem with extra steps.

Confidence Is a Routing Decision, Not a Footnote

A confidence score only earns its place if something happens differently below a threshold.

if (row.confidence < 0.6) {
  row.status = "needs_human_review";
} else {
  row.status = "classified";
}
Enter fullscreen mode Exit fullscreen mode

Below threshold, the row waits for a person instead of getting auto-tagged, auto-routed to an owner, or auto-counted into a theme total. A model that is unsure and a model that is confident should never look the same in the data. Silent low-confidence auto-action is how a pipeline quietly poisons a report weeks later, once nobody remembers which rows were guesses.

Re-Runs Have to Be Idempotent

You will re-classify the same batch more than once — a prompt improves, a category gets added, someone reruns last week's data by mistake. If the re-run is not idempotent, tags duplicate and downstream actions, like a Slack alert or an owner assignment, fire twice for the same response.

const key = `${response_id}:${classifier_version}`;

if (alreadyClassified.has(key)) skip();
else {
  classify(response_id);
  record(key);
}
Enter fullscreen mode Exit fullscreen mode

Version the classifier, not just the response, so a genuine re-classification stays a deliberate, traceable event rather than a side effect of re-running a script.

Where the Inference Actually Runs

None of this needs a server-side LLM. Run against FORMLOVA response data, this classification happens in your own connected AI client — during an MCP session, using your own model — not inside FORMLOVA's infrastructure. FORMLOVA exposes the structured response data and stores the resulting fields; it does not classify your responses for you on a server.

The full context for voice-of-customer analysis, including which sources to mix and how to report findings, is here: Voice of Customer Analysis with AI.

Top comments (0)