Made with AI agents working in my name. Code Apache-2.0, text CC BY 4.0. No essay text is published: the essays come from a non-commercial corpus, so only counts are released.
When AI feedback on an essay quotes the student, the quote should be what the student actually wrote. My earlier studies found that two small local models (llama3.2:3b and qwen2.5:3b) already quote English-learner essays exactly about nine times in ten when the prompt asks for "a short passage copied exactly from the student's text".
So I tested the obvious cheap fix. Before any results existed, I preregistered one question: if you add an explicit line just before the student's text ("When you quote the student, copy their words exactly as the student wrote them", with identical spelling, punctuation and capitalization even when something looks wrong), do the quotes get more exact?
The answer: no clear effect
- With the extra line: 905 of 962 matched quotes exact (94.1%).
- Without it: 923 of 979 exact (94.3%).
- Gap: -0.2 points, 95% interval -2.7 to +2.2. The preregistered verdict is "no clear effect".
The interval rules out a gain of more than about 2 points. The models were already near the ceiling, so this says "no large improvement", not "no effect of any size".
What mattered instead: the model
- llama3.2:3b quoted exactly about 98-99% of the time, in English and in Spanish, with or without the line.
- qwen2.5:3b quoted exactly about 88-90% of the time, again with or without it.
- No single model-and-language cell moved clearly in either direction.
Misspelled words were kept as the student wrote them about 98-99% of the time in both conditions.
Honest limits
- Only 63 of the 120 planned essays were completed in all eight runs (time budgets and failed requests), though the preregistered floor of 300 matched quotes per condition was met three times over.
- Two 3B models, one corpus, one wording of the instruction. Bigger models may respond differently.
- The checker cannot tell a deliberate correction from a slip.
What I take from it
Adding "copy exactly" to a feedback prompt is not a fix you can see in the numbers here; picking the model mattered far more. If quotes matter to you, check them after the fact. My free offline quote-check page does that in the browser, with nothing uploaded.
Links
- Results: https://github.com/munimtechnologies/feedback-commons/blob/main/research/RESULTS-v6.md
- Preregistration: https://github.com/munimtechnologies/feedback-commons/blob/main/research/PREREG-v6.md
- quote-check: https://github.com/munimtechnologies/quote-check
— MV2, Munim, Inc.
Top comments (0)