On 18 public Ruff issues, a small local model and keyword rules each matched 11 saved labels. On two generated product images, changing how the same model's answers were scored changed every image-only decision. Both results needed more explanation than a headline score.
AI disclosure: An AI agent ran the experiments, generated the test images and drafted this article. I chose the photo/text conflict-review rule during our discussion.
AI-generated test images. Two designed scenes, not real product-inspection evidence.
What was tested
This used mjdileep/OpenJev, an independent Jev-style project, rather than an official open release of TypeSafe's Jev. OpenJev turns supplied text or images into category choices. It does not create the images.
The model was Qwen3.5-0.8B, in 4-bit form, running locally through MLX on an Apple M3 Pro with 18 GiB memory and Python 3.12.11. The image tests also used MLX-VLM 0.7.8. No model training was performed. Image generation used a separate built-in image-generation service.
Text: the original tie
The saved data had 24 Ruff reports: six development examples and 18 evaluation examples, with six per category. Reference categories came from bug, documentation and wish, mapping wish to feature. The sample followed GitHub's returned order within each label and included closed reports. It was not a random sample of incoming issues.
The rules searched titles for documentation words first, then feature-request words, otherwise choosing bug. OpenJev received the title and up to 1,600 body characters, without the reference label.
Both methods matched 11/18 labels: six bugs, two documentation reports and three features. Nine reports matched both methods, two matched only the rules, two only the model, and five neither. Repeating the model run preserved all 18 choices. Its median request time in that repeat was 0.513 seconds, excluding loading and warm-up; this was one local timing observation, not a speed benchmark.
The input difference mattered, so a later exploratory run gave both methods the same input:
Measured label matches on the same 18 already-inspected reports. The follow-up reused this sample; it was not a new blind evaluation.
More text helped this model and hurt these rules. A documentation link in a bug report can trigger a rule that gives documentation words priority. That is a property of this rule recipe; it does not show that rules generally fail with longer text.
A question before judging the score
Ruff #21099 says that ruff analyze graph seems to skip notebooks. Is that a bug or a feature request?
My initial reading was feature: notebook support is missing. The saved wish label agrees. But the maintainer discussion found code explicitly skipping notebooks. Removing that filter made notebook files appear, though their imports were still missing. The intended behaviour matters before deciding which category is right.
Ruff #28493 gives another reason to inspect labels. It describes a supposedly safe fix creating a type-checker error. The maintainer explains Ruff's runtime-oriented safety definition and accepts documenting the limitation. A documentation label fits that resolution, while a classifier reading the original report could reasonably choose bug.
These discussions were inspected after evaluation. The original reference answers were not rewritten. The measured quantity remains agreement with repository labels, rather than independently checked accuracy.
Images: make your prediction first
The pictures above show a red mug with no visible damage and a blue bowl with a visible chip and crack. Each image was classified on three attributes: object, colour and condition. The agent visually inspected the generated outputs before inference; these are not independent human labels.
Each scene was tested with image only, a matching description, a conflicting description, and a text-only control. The instruction said to judge the image when supplied. For text-only controls, the expected answer followed the description.
The default image-only calls got 0/6 attributes right. A smaller batch did not change that result. The same model's direct descriptions, using a different prompt through MLX-VLM, correctly described the red mug and chipped blue bowl.
That prompted a scoring check. With the same image-only inputs, questions, weights and batch size of one, switching from score_mode="full" to score_mode="binary" matched 6/6 attributes.
AI-generated scenes with an animated replay of measured outputs. This GIF is not live inference. Six attributes across two pictures do not establish general accuracy. The scoring diagnostic was chosen after the initial failures.
Why did the ranking reverse?
The default scoring compares the model's support for “yes” for each candidate. Binary scoring instead compares “yes” against “no” for that candidate.
For the red-mug colour question, the saved default run produced approximately: Default scoring selects blue because its “yes” value is larger. Comparing only yes and no gives a different ranking: The red candidate's “yes” is small, but its “no” is smaller still. These are response scores, not calibrated chances that a category is correct. We have not established which other possible response words explain the difference.Inspect the scores behind the colour reversal
yes / (yes + no)
red ≈ 0.9883
blue ≈ 0.5000
The caption tests were also imperfect: default scoring matched 0/6 image attributes with accurate captions and 1/6 with conflicting captions. Text-only controls matched 5/6 description attributes. Captions were not separately rerun under binary scoring, so the image-only result does not establish resistance to misleading text.
What should happen when the photo and text disagree?
My choice from the discussion was: flag the conflict for review. Show the conflicting details before accepting a category.
A blue bowl paired with “an intact red mug” should not silently become a settled product record. But disagreement alone does not prove which source is wrong: the photo, description or model could be mistaken. This is a chosen review rule, not an implemented or validated conflict detector. The executed test still forces category choices.
What these results support
The text comparison gives no reason to replace the simple rules for this small task. The image example shows that candidate scoring can reverse decisions even when the underlying model can describe the scenes. It does not establish a general OpenJev bug or show that binary scoring is always better.
The limits are substantial: one model, one repository, 18 previously inspected issue reports and two designed images. The image attributes change together, the choices have no “unknown” option, and no real-photo or independent human evaluation was performed. Source issues may also have appeared in training data. No larger-model comparison or production reliability claim follows.
The useful questions were what the labels describe, what information reaches each method, and how a response score becomes a decision. Those questions explain more than the score alone.
Top comments (0)