Vendors love to demo on clean examples. The best defence is your own test set, picked fairly from real history. That set is the backbone of any honest attempt to audit a business workflow with real data before buying AI.
Here is a small script that builds one from a ticket or enquiry export.
Input
Any CSV with an id, a category and a few signals of trouble. For example:
id,category,reopened,replies,minutes_to_first_reply
1001,refund,0,2,14
1002,delivery,1,6,310
1003,refund,0,1,9
The script
import csv, random
from collections import defaultdict
random.seed(7) # repeatable sample
def bucket(row):
if int(row["reopened"]) or int(row["replies"]) >= 5:
return "awkward"
if int(row["minutes_to_first_reply"]) > 240:
return "slow"
return "ordinary"
def sample(path, per_bucket=None):
per_bucket = per_bucket or {"ordinary": 10, "awkward": 6, "slow": 4}
groups = defaultdict(list)
with open(path, newline="") as f:
for row in csv.DictReader(f):
groups[(bucket(row), row["category"])].append(row)
picked = []
for b, n in per_bucket.items():
keys = [k for k in groups if k[0] == b]
i = 0
while n > 0 and any(groups[k] for k in keys):
k = keys[i % len(keys)]
if groups[k]:
picked.append(groups[k].pop(random.randrange(len(groups[k]))) | {"bucket": b})
n -= 1
i += 1
return picked
if __name__ == "__main__":
rows = sample("tickets.csv")
with open("eval_set.csv", "w", newline="") as f:
w = csv.DictWriter(f, fieldnames=list(rows[0].keys()))
w.writeheader(); w.writerows(rows)
print(len(rows), "cases written to eval_set.csv")
Why it is built this way
- Round-robin across categories so one busy category does not crowd out the rest.
- Separate buckets for awkward and slow cases, because that is where tools fail.
- A fixed seed so you and the vendor test on the same set.
What to do with the set
Add an "expected outcome" column by hand. Then run every tool you are considering on the same cases, in draft mode, and score the results. Keep the set private so nobody tunes against it.
I work at NxFlowAI, and we use a set like this after our 72-hour audit. For more posts on evaluation and workflows, see the NxFlowAI blog index.
Top comments (0)