Disclosure: we build CompsAPI, the sold-price API whose filter is measured in this post. It was drafted with AI assistance, and every figure below comes from CompsAPI's published test notes. The labels in the test were made by us, which is a limit I come back to at the end.
"The model is 92% accurate" is a sentence that sounds like a result and tells you almost nothing. This post takes one real example, a classifier that labels marketplace sales as "a normal item" or "junk", and shows what the published test numbers say once you split the one percentage into the parts it hides.
The task
Sold-price data from resale marketplaces is full of rows that are not the thing you searched for: lots, parts, accessories, empty boxes, mystery packs, replicas and digital codes. A median over all of them can describe nothing you could actually buy.
CompsAPI's classifier gives every sale one label (item, lot, parts, accessory, box_only, mystery, replica, digital or other). Everything except item counts as junk. Nothing is deleted: with ai=1 every row keeps its label, and with exclude=ai_junk the junk rows are left out of the results and the median.
How it was tested
Three choices in the method are worth copying if you ever evaluate a classifier yourself:
- Two independent labels per listing. Each test listing was labelled twice independently, and the disagreements were settled by review. Without that, you are measuring your own labelling mood as much as the model.
- Listings the model never trained on. None of the test listings were used to train the filter. For eBay, the 690 test listings came from 20 searches the filter had never seen, which tests generalisation, not memory.
- eBay measured separately from the other marketplaces. The junk rate is very different between them, as the next section shows.
The results
| Test set | Listings | Junk in the set | Agreement with labels |
|---|---|---|---|
| Marketplaces (Reverb, Mercari, Poshmark and others) | 600 | 51 | 97.7% |
| eBay, unseen searches | 690 | 277 | about 92% |
Now split the eBay figure into its two kinds of mistake:
- Real items wrongly removed: 24 of 413, which is 5.8%.
- Junk wrongly kept: 30 of 277, which is 10.8%.
On the marketplaces: 8 of 549 real items were wrongly removed (1.5%), and 6 junk rows were wrongly kept.
So "92%" is really "removes about 1 in 17 real items and lets through about 1 in 9 junk rows". Those are different problems, and they hurt different uses.
Why one accuracy number hides the base rate
Here is the part that changed how I read these numbers. Compare the filter with a classifier that does nothing and calls every listing a real item:
| Test set | Do-nothing baseline | Filter |
|---|---|---|
| Marketplaces (549 real of 600) | 91.5% | 97.7% |
| eBay (413 real of 690) | 59.9% | about 92% |
The do-nothing figures are my arithmetic from the published counts, not a number the test reports. They show why the same word, "accuracy", means different things on the two sets. On the marketplaces, junk is rare (about 8.5% of the sample), so a lazy baseline already scores 91.5% and the filter's 97.7% is a more modest step up. On eBay, about 40% of the test listings were junk, so the baseline is poor and the filter's job is much bigger.
If you evaluate your own classifier, report the base rate and both error types next to the headline number.
Which mistake costs more?
That depends on what you do with the output. This is reasoning, not a measurement:
- If you compute a median, a few removed real items thin the sample a little, but missed junk can pull the median away from the true price. For that use, the 10.8% miss rate on eBay is the figure to watch.
- If you are hunting for individual deals, a real item wrongly removed is a deal you never see, so the 5.8% matters more.
What the filter is bad at
The test notes are specific about this, which is the useful part:
- By kind of junk, across the test: lots, 25 of 26 removed. Parts, 13 of 13. Mystery boxes and packs, 30 of 32. Listings that are not a product, such as services, 18 of 18. Accessories, only 35 of 41.
- Bundles. A console sold with its own dock and controller should be kept, and usually is, but not always: 2 of 23 higher-priced removed listings for Xbox Series X, 2 of 21 for Nintendo Switch OLED, 1 of 18 for PS5 and 5 of 68 for PlayStation 4 Slim were single consoles.
- Vague titles. Very short titles, one or two words, are sometimes left out as unclear.
- Multipacks sold as one product, such as a 2 pack or a 6 pack, are sometimes treated as lots.
- "Body only" cameras are sometimes treated as parts.
- Real lots that stay in. A console with games or two controllers can survive the filter, usually when the listing says so in another language or avoids a word like "bundle".
- Counterfeits. It labels a listing a replica only if the listing says so. A fake sold as real is outside what it can see.
Run your own check in 20 lines
You should not take a filter's word for it. This script pulls one search with labels attached, counts what the model thinks is in it, and prints the rows it was least sure about, so you can judge those yourself:
import os
import sys
from collections import Counter
import requests
resp = requests.get(
"https://api.compsapi.com/v1/sold",
params={"q": " ".join(sys.argv[1:]), "ai": 1},
headers={"X-API-Key": os.environ["COMPSAPI_KEY"]},
timeout=30,
)
resp.raise_for_status()
rows = [r for r in resp.json().get("items", []) if r.get("ai")]
print(Counter(r["ai"]["kind"] for r in rows))
print("\nLeast sure rows:")
for r in sorted(rows, key=lambda r: r["ai"]["confidence"])[:15]:
ai = r["ai"]
print(f'{ai["confidence"]:.2f} {ai["kind"]:<10} ${r["price"]:>9,.2f} {r["title"][:70]}')
Run it as python check.py "nintendo switch oled" and compare the low-confidence rows with your own eye. ai=1 removes nothing, so it is safe to try on any search. The script only prints rows that came back with a label. A very recent sale can arrive before it has one.
Limits of these numbers
- The labels are ours. The test measures agreement with labels made by the same team that built the filter, double-checked and reviewed, but not by an outside party.
- Small samples. 20 eBay searches is not all of eBay. Several marketplaces had fewer than 50 test listings, and a few had no junk at all in the sample, so they get no accuracy figure.
- Agreement is not truth. It tells you how often the model matches a reviewed label, not whether the remaining rows match the exact model you meant.
-
I did not run the script above against a live key. It follows the documented fields (
ai.kind,ai.confidence,price,title), but treat it as a starting point.
CompsAPI is not affiliated with eBay or the other marketplaces mentioned.
Try it
A free key gives you 100 requests a month, enough to run the check on a few searches you know well: get one here. If you find a listing the filter gets wrong, send it to us in the comments.
Top comments (0)