One setup marked four defective items as normal. Another marked seven as normal. The first sounds safer until you see the other half of the result: it also marked 69 of 80 normal items as defective. The second did that to 33.
Those decisions came from the same 160 inspection photos. They make a more useful comparison than the two generated images that started this experiment.
In the first article, changing OpenJev's scoring reversed six attribute decisions across a red mug and a blue bowl. This follow-up asks whether that change holds up on public inspection images, and compares it with OpenAI's Decisions API.
AI assistance: An AI agent used to perform experiments and help to draft article.
Three setups, with two different kinds of comparison
The local model was Qwen3.5-0.8B in 4-bit form, used through the independent mjdileep/OpenJev project on an Apple M3 Pro. This project is separate from TypeSafe's Jev. OpenJev supplied the classification layer; it did not generate these photos.
Each local candidate receives its own yes/no evaluation. For our two choices, the questions amount to whether the item is normal and whether it is defective.
OpenJev full scoring ranks those candidates using support for the “yes” response token. Each support is divided by the total across the offered categories, so the reported numbers add up to one.
OpenJev binary rescoring instead uses yes / (yes + no) for each candidate, then applies that same final division across the categories. Here, “binary” means comparing those two response tokens. The change is arithmetic on the same saved model scores. We did not retrain the model or run a second optimized binary inference for this comparison.
OpenAI Decisions, using gpt-6-luna, received a choice question with the same two labels and descriptions. The official API documentation describes image inputs and answers chosen from a fixed set with a distribution over the options. We used that interface, rather than asking for a written explanation.
The first comparison isolates a scoring rule within the same local outputs. The second compares complete systems: their weights, image processing and serving differ. A result between OpenJev and OpenAI cannot be attributed to the scoring formula alone.
From two designed scenes to 200 real images
The photos came from MVTec AD, through Voxel51's public conversion at a fixed revision. They cover bottles, capsules, hazelnuts, metal nuts and screws.
For each category, we selected 20 normal and 20 defective images. Defective examples were sampled across the available defect types. Forty images were set aside for development checks; the reported main comparison uses the remaining 160, with 80 normal and 80 defective items. These are our own divisions of the source test images, not the dataset's official benchmark procedure.
The file list, labels and questions were fixed before inference. No prompt was changed after the development results. Both systems received identical image bytes: RGB images resized without changing aspect ratio to at most 768 pixels on the longest side, then saved as JPEG. Each model received the item category and inspection question, but not the image's defect label, subtype or filename.
No model was trained. No example of a normal item was supplied as a reference photo.
Here, a missed defect means a dataset-labelled defective item marked normal. A false alarm means a dataset-labelled normal item marked defective. “Correct” means agreement with those supplied labels; we did not independently adjudicate them.
The overall score, and what sits underneath it
| Setup | Correct out of 160 | Defects marked normal, out of 80 | Normal items flagged, out of 80 |
|---|---|---|---|
| OpenJev full | 76 — 47.5% | 25 | 59 |
| OpenJev binary rescore | 87 — 54.4% | 4 | 69 |
| OpenAI Decisions | 120 — 75.0% | 7 | 33 |
Measured results. Accuracy lines show approximate 95% intervals: 39.9–55.2% for full scoring, 46.6–61.9% for binary rescoring and 67.8–81.1% for Decisions. They assume independent images; unknown relationships between photographed items limit that assumption.
OpenAI had the highest overall agreement in this test. Binary rescoring marked fewer defects normal, while producing substantially more false alarms.
Its 54.4% result also needs a simple reference: always answering normal would score 50% on this balanced sample. Always answering defective would do the same. Binary rescoring called every bottle, capsule and hazelnut defective. Much of its success at finding defects came with a broad tendency to flag items.
Across all five categories, it changed 33 decisions from the full-scoring result. Twenty-two changes corrected an error; eleven introduced one. The larger sample replaces the earlier apparently perfect result with a mixed one.
One photo shows what the formula changes
This bottle is labelled broken_small in the dataset. Full scoring selected normal; binary rescoring selected defective.
Two actual evaluation images, selected after evaluation. This is a replay of saved decisions, not live inference. The scoring change corrects the defective bottle on the left and mislabels the normal bottle on the right. Images: MVTec AD, Bergmann and colleagues, via Voxel51; CC BY-NC-SA 4.0; resized.
The saved local scores were:
| Candidate | Yes score | No score | Yes / (yes + no) |
|---|---|---|---|
| Normal | 0.025154 | 0.019590 | 0.5622 |
| Defective | 0.024614 | 0.010261 | 0.7058 |
Full scoring favours normal because 0.025154 is slightly larger than 0.024614. Binary rescoring favours defective because its “no” score is much smaller relative to its “yes” score.
There is a second normalisation across the two categories. The final normal/defective scores become approximately 0.5054 / 0.4946 under full scoring and 0.4434 / 0.5566 under binary rescoring.
The image, prompt and model outputs did not change. The ranking rule did. These scores are not established as real-world probabilities of being correct.
The opposite also happened. On a dataset-labelled normal bottle, full scoring selected normal at 0.5679; binary rescoring selected defective at 0.5288. A scoring change that fixes one case can break another.
Illustrations selected after evaluation, using the first case matching each stated error pattern. They are not a random sample. The full-sample counts above include every evaluation image. Photos: MVTec AD, Bergmann and colleagues, via Voxel51; CC BY-NC-SA 4.0; resized.
The category breakdown changes the picture again
| Category | OpenJev full | Binary rescore | OpenAI Decisions |
|---|---|---|---|
| Bottle | 12/32 | 16/32 | 22/32 |
| Capsule | 15/32 | 16/32 | 29/32 |
| Hazelnut | 17/32 | 16/32 | 29/32 |
| Metal nut | 13/32 | 19/32 | 16/32 |
| Screw | 19/32 | 20/32 | 24/32 |
OpenAI matched 29 of 32 capsule labels and 29 of 32 hazelnut labels. On metal nuts, it called all 32 items defective, leaving it at 16/32. Binary rescoring did better on that category, at 19/32.
The 75% overall figure therefore hides a category where the hosted system failed to distinguish the normal items. A deployment focused on metal nuts would need to examine that result directly.
This setup also has an important gap: some anomalies depend on knowing the normal shape, marking or orientation. We supplied no normal reference photo. That may contribute to errors, but this experiment does not establish their cause.
Put the priorities into the comparison
During the discussion, I asked to keep both kinds of error visible. One method catching more defects is useful information; the extra normal items it flags are part of the same result.
If every predicted defect were sent for review, binary rescoring would queue 145 of the 160 items. OpenAI would queue 106. That is 39 additional reviews for three additional correctly flagged defects; the other 36 additional flags are normal items. This is an illustration of a possible workflow, not an executed review process.
Left: measured error counts. Right: arithmetic on those counts, with each false alarm assigned one penalty unit and each missed defect assigned a variable penalty. The dashed horizontal line is the always-defective reference. This illustration excludes inference expense and the cost of reviewing correctly flagged defects; these are not measured business costs.
For that deliberately simple error model:
Full scoring penalty = 25 × missed-defect penalty + 59
Binary penalty = 4 × missed-defect penalty + 69
OpenAI penalty = 7 × missed-defect penalty + 33
Binary rescoring and OpenAI tie when a missed defect carries 12 times the penalty of a false alarm. Below that ratio, OpenAI has the smaller weighted error on this sample; above it, binary rescoring does.
That crossover depends on the assumptions. If every flagged item carries a review cost, including true defects, the extra 39 flags move the corresponding crossover to 13. A different defect frequency would also change the comparison. Neither number is a production recommendation.
Compare against flagging everything, too. Always calling an item defective produces zero missed defects and 80 false alarms on this sample. Under the error-only illustration, that is a penalty of 80 at every ratio. At ratio 12, both binary rescoring and OpenAI score 117. A method can therefore beat another model under chosen weights while still losing to a constant rule. That is another reason to keep the reference visible rather than turn the crossover into a deployment claim.
The penalty chart is arithmetic on saved counts; it does not include any additional model calls.
Captions exposed another weakness
We also selected 20 evaluation images before inference: two normal and two defective items from each category. Each was tested with a matching caption, a conflicting caption, conflicting text without the image, and reversed choice order. Captions were controlled statements such as “this item is normal,” not naturally collected product descriptions.
These are repeated conditions on 20 images, not 80 new independent examples. Expected answers follow the image when present. For the text-only control, they follow the text.
On this subset, OpenAI matched 15/20 image-only labels, 18/20 with matching captions and 13/20 with conflicting captions. All three setups separately matched the 20 text-only descriptions.
The default local rule produced a surprising result: conflicting captions increased its matches from 7/20 to 18/20. Binary rescoring stayed at 10/20 under that change. This is an unresolved observation about these prompts and scores. It does not establish that incorrect captions improve classification, or that the model reliably recognised a conflict.
Reversing the choices changed two OpenAI decisions, although its total stayed at 15/20. After seeing that, we added one unchanged-request repeat on all 20 images. That repeat preserved every choice for all methods. The order changes were therefore not also seen in this single repeat control. One repeat cannot show that answers will always repeat, and the added control was chosen after inspecting the first results.
Flagging disagreement still needs a dependable image decision
My earlier choice was to flag photo/text disagreement for review. We tested a simple version: compare the image-only decision with the supplied caption label, and flag a mismatch.
Each of the 20 images received one matching and one conflicting caption assignment. Of the 20 intended conflicts, OpenAI's image-only decisions would flag 15 and miss five. They would also trigger five unnecessary reviews on matching captions. Binary rescoring's counts were ten, ten and ten; full scoring's were seven, thirteen and thirteen.
A wrong image decision can agree with a wrong caption. Agreement alone cannot tell us that the record is right. This rule was evaluated on 40 controlled caption scenarios built from 20 images; it is not a validated general conflict detector.
What the test can support
This is a small, balanced study across five selected categories. Its main result uses 160 images, not all 200, and repeated captions do not enlarge that count. The pretrained models may have encountered this public benchmark during training. Exact duplicate images were excluded, but identities of the physical items were unavailable. Resizing can hide small defects, and we used the supplied labels without independent human review.
The local model, question wording and preprocessing were fixed. Different weights, reference examples, prompts or thresholds could produce different results; they were not tested here. This is also a comparison of a local setup with a hosted service, not a claim about every OpenJev configuration or TypeSafe's Jev.
The hosted study's 300 successful calls, including development, diagnostics and the repeat control, reported 214,668 input tokens. This excludes the earlier connection checks and synthetic-image pilot.
Ruff remains a separate text-routing experiment. Its 18 issue reports and repository-label agreement are not included in these image counts.
For these images, OpenAI had better overall agreement and fewer false alarms. Binary rescoring missed fewer defects while flagging most normal items. The useful comparison keeps those outcomes together—and keeps the actual mistakes visible beside the scores.
The run used OpenJev commit MVTec AD photographs are by Paul Bergmann and colleagues and were obtained through Voxel51's conversion. Their licence is CC BY-NC-SA 4.0, including attribution, non-commercial use and share-alike requirements. Photos were resized and combined with labels for the figures. The mirror revision was pinned; we did not verify the publisher's original archive checksum. See the dataset publisher and conversion card for provenance.Source and image notes
501da7e6b59c71e1d184a0eb37e36f8cc5bc6354, Qwen3.5-0.8B 4-bit revision da28692, MLX-VLM 0.7.8 and the returned OpenAI model name gpt-6-luna. Both systems received the same inspection instruction, candidate descriptions and prepared image bytes. Selection used seed 20261017; prompts, selected file identifiers, hashes, outputs and failures were recorded before summarising results. Reusing the hosted model name later does not guarantee identical underlying weights.
Top comments (0)