On 18 public Ruff issues, a small local model and keyword rules each matched 11 saved labels. On two generated product images, changing how the same model's answers were scored changed every image-only decision. The first result raised a question about the reference labels. The second raised a question about how model responses become category choices.
AI assistance: An AI agent used to perform experiments and help to draft article.
What was tested
This used mjdileep/OpenJev, an independent Jev-style project, rather than an official open release of TypeSafe's Jev. OpenJev turns supplied text or images into category choices. It does not create the images.
The model was Qwen3.5-0.8B, in 4-bit form, running locally through MLX on an Apple M3 Pro with 18 GiB memory and Python 3.12.11. The image tests also used MLX-VLM 0.7.8. No model training was performed. Image generation used a separate built-in image-generation service.
Text: a tie against repository labels
The saved data had 24 Ruff reports: six development examples and 18 evaluation examples, with six per category. Reference categories came from bug, documentation and wish, mapping wish to feature. The sample followed GitHub's returned order within each label and included closed reports. It was not a random sample of incoming issues.
The rules searched titles for documentation words first, then feature-request words, otherwise choosing bug. OpenJev received the title and up to 1,600 body characters, without the reference label.
Both methods matched 11/18 labels: six bugs, two documentation reports and three features. Nine reports matched both methods, two matched only the rules, two only the model, and five neither. Repeating the model run preserved all 18 choices. Its median request time in that repeat was 0.513 seconds, excluding loading and warm-up; this was one local timing observation, not a speed benchmark.
The input difference mattered, so a later exploratory run gave both methods the same input:
Measured label matches on the same 18 already-inspected reports. The follow-up reused this sample; it was not a new blind evaluation.
More text helped this model and hurt these rules. A documentation link in a bug report can trigger a rule that gives documentation words priority. That is a property of this rule recipe; it does not show that rules generally fail with longer text.
A question before judging the score
Ruff #21099 says that ruff analyze graph seems to skip notebooks. Is that a bug or a feature request?
My initial reading was feature: notebook support is missing. The saved wish label agrees. But the maintainer discussion found code explicitly skipping notebooks. Removing that filter made notebook files appear, though their imports were still missing. The intended behaviour matters before deciding which category is right.
Ruff #28493 gives another reason to inspect labels. It describes a supposedly safe fix creating a type-checker error. The maintainer explains Ruff's runtime-oriented safety definition and accepts documenting the limitation. A documentation label fits that resolution, while a classifier reading the original report could reasonably choose bug.
These discussions were inspected after evaluation. The original reference answers were not rewritten. The measured quantity remains agreement with repository labels, rather than independently checked accuracy.
Images: make your prediction first
The pictures above show a red mug with no visible damage and a blue bowl with a visible chip and crack. Each image was classified on three attributes: object, colour and condition. The agent visually inspected the generated outputs before inference; these are not independent human labels.
Each scene was tested with image only, a matching description, a conflicting description, and a text-only control. The instruction said to judge the image when supplied. For text-only controls, the expected answer followed the description.
The default image-only calls got 0/6 attributes right. A smaller batch did not change that result. The same model's direct descriptions, using a different prompt through MLX-VLM, correctly described the red mug and chipped blue bowl.
That prompted a scoring check. With the same image-only inputs, questions, weights and batch size of one, switching from score_mode="full" to score_mode="binary" matched 6/6 attributes. These six decisions describe only two images. Object, colour and condition change together, so they are not six independent examples.
AI-generated scenes with an animated replay of measured outputs. This GIF is not live inference. Six attributes across two pictures do not establish general accuracy. The scoring diagnostic was chosen after the initial failures.
Why did the ranking reverse?
OpenJev asks about each candidate separately. For the colour question, “red” and “blue” each receive their own yes/no evaluation. The default scoring uses the model's support for the “yes” token. Binary scoring uses yes / (yes + no) for that candidate. The decision code then normalises the candidate supports so the offered categories sum to one. Those are two different steps.
For the red-mug colour question, the saved default run produced approximately: Default scoring selects blue because its “yes” value is larger. Comparing only yes and no gives a different ranking: The red candidate's “yes” is small, but its “no” is smaller still. Among those two response tokens, red has stronger support for yes. The default rule instead rewards blue's much larger absolute yes score. There is another important distinction: 0.9883 is the red candidate's yes/no support, not its final probability among the colour choices. Normalising the saved supports across red and blue gives: These calculations reuse the initial saved call. They are not the scores from a new model run. Recomputing all six image-only choices from those saved yes/no values changes agreement from 0/6 to 6/6, without changing an image, prompt or model output. This separates the arithmetic effect from differences between separate inference calls. The separate batch-size-one checks reported above also matched 6/6 under binary scoring. The saved values expose a further clue: yes and no together account for about 0.53% of the red candidate's response-token probability, compared with 84.16% for blue. Binary scoring discards that difference in total mass. It can therefore give strong relative support to yes even when both yes and no are unlikely responses. We have not identified which other response tokens receive the remaining mass, or established why the prompts produce such different totals. Neither the normalised category scores nor the yes/no supports are calibrated chances of being correct. A larger score does not by itself establish a more reliable decision.Inspect the scores behind the colour reversal
yes / (yes + no)
red ≈ 0.9883
blue ≈ 0.5000
Default: red / (red + blue)
≈ 0.005198 / (0.005198 + 0.420787)
≈ 0.0122
Binary: red / (red + blue)
≈ 0.9883 / (0.9883 + 0.5000)
≈ 0.6640
The caption tests were also imperfect: default scoring matched 0/6 image attributes with accurate captions and 1/6 with conflicting captions. Text-only controls matched 5/6 description attributes. Captions were not separately rerun under binary scoring, so the image-only result does not establish resistance to misleading text.
My choice for photo/text disagreement was to flag the conflict for review. That remains a desired behaviour: this experiment forces category choices and does not implement or validate a conflict detector. Disagreement alone cannot tell us whether the photo, description or model is wrong.
What these results support
The text comparison gives no reason to replace the simple rules for this small task. The image example shows that candidate scoring can reverse decisions even when the underlying model can describe the scenes. It does not establish a general OpenJev bug or show that binary scoring is always better.
The limits are substantial: one model, one repository, 18 previously inspected issue reports and two designed images. The image attributes change together, the choices have no “unknown” option, and no real-photo or independent human evaluation was performed. Source issues may also have appeared in training data. No larger-model comparison or production reliability claim follows.
The text study shows why agreement depends on what the reference labels describe and which input each method receives. The image study isolates a different issue: the rule that converts response scores into category choices can reverse every decision in a small example. It leaves open whether that behaviour repeats on fresh images.
The follow-up is now live: OpenJev vs OpenAI Decisions: fewer missed defects, more false alarms. It compares 160 public inspection photos, with charts, worked scores and examples of both kinds of error.
Top comments (0)