DEV Community

Ujjwal Dubey
Ujjwal Dubey

Posted on

Build a Stratified Sample of Real Cases From a Ticket Export Before You Evaluate Any AI Tool

Vendors love to demo on clean examples. The best defence is your own test set, picked fairly from real history. That set is the backbone of any honest attempt to audit a business workflow with real data before buying AI.

Here is a small script that builds one from a ticket or enquiry export.

Input

Any CSV with an id, a category and a few signals of trouble. For example:

id,category,reopened,replies,minutes_to_first_reply
1001,refund,0,2,14
1002,delivery,1,6,310
1003,refund,0,1,9
Enter fullscreen mode Exit fullscreen mode

The script

import csv, random
from collections import defaultdict

random.seed(7)  # repeatable sample

def bucket(row):
    if int(row["reopened"]) or int(row["replies"]) >= 5:
        return "awkward"
    if int(row["minutes_to_first_reply"]) > 240:
        return "slow"
    return "ordinary"

def sample(path, per_bucket=None):
    per_bucket = per_bucket or {"ordinary": 10, "awkward": 6, "slow": 4}
    groups = defaultdict(list)
    with open(path, newline="") as f:
        for row in csv.DictReader(f):
            groups[(bucket(row), row["category"])].append(row)
    picked = []
    for b, n in per_bucket.items():
        keys = [k for k in groups if k[0] == b]
        i = 0
        while n > 0 and any(groups[k] for k in keys):
            k = keys[i % len(keys)]
            if groups[k]:
                picked.append(groups[k].pop(random.randrange(len(groups[k]))) | {"bucket": b})
                n -= 1
            i += 1
    return picked

if __name__ == "__main__":
    rows = sample("tickets.csv")
    with open("eval_set.csv", "w", newline="") as f:
        w = csv.DictWriter(f, fieldnames=list(rows[0].keys()))
        w.writeheader(); w.writerows(rows)
    print(len(rows), "cases written to eval_set.csv")
Enter fullscreen mode Exit fullscreen mode

Why it is built this way

  • Round-robin across categories so one busy category does not crowd out the rest.
  • Separate buckets for awkward and slow cases, because that is where tools fail.
  • A fixed seed so you and the vendor test on the same set.

What to do with the set

Add an "expected outcome" column by hand. Then run every tool you are considering on the same cases, in draft mode, and score the results. Keep the set private so nobody tunes against it.

I work at NxFlowAI, and we use a set like this after our 72-hour audit. For more posts on evaluation and workflows, see the NxFlowAI blog index.

Top comments (0)