Claude knows what a reverse-scored item is. If you ask it "explain reverse scoring in psychometrics," it'll give you a textbook definition — how negatively worded items are scored in the opposite direction to prevent response bias, how a 5 on "I am often unhappy" should be scored as a 1 on a wellbeing scale.
But knowing the concept and applying it correctly when building a form are different skills. Here's what happened when I asked Claude to build a PHQ-9 depression screening form with proper scoring.
The PHQ-9 Challenge
The PHQ-9 is a 9-item depression screening tool. Each item is scored 0-3 (Not at all = 0, Several days = 1, More than half the days = 2, Nearly every day = 3). Total score ranges 0-27. Higher = more depressive symptoms.
Simple enough. No reverse items (unlike many wellbeing scales). The challenge was getting Claude to set up the scoring correctly on each field.
What Claude Got Right
Claude nailed the field structure. Nine radio fields, each with four options, each option scored 0-3:
field_add --app <appId> --id q1 --type radio --title "Little interest or pleasure in doing things" --options "Not at all:0,Several days:1,More than half:2,Nearly every day:3"
It used label:score format correctly. The scoring was embedded in the options string. When a respondent selects "Several days," the system records a score of 1. Claude understood the format from the schema description:
options: z.string().optional().describe('Options list, comma-separated, supports label:score'),
That one description — "supports label:score" — was enough. Claude figured out the rest.
What Claude Got Wrong
The problem wasn't the scoring. It was the total. After building the form, I asked Claude: "What's the total score range?"
Claude said: "0 to 27, with higher scores indicating more severe depressive symptoms."
Correct. But the form Claude built couldn't compute that total. At the time, the MCP tools could set per-option scores, but couldn't configure the scale-level scoring — the part that says "sum all 9 items and compare to cutoffs (5=mild, 10=moderate, 15=moderately severe, 20=severe)."
That configuration lives in the Scale module, which the MCP server didn't expose at the time. So Claude built a form that collects the right data, but it couldn't tell the respondent "your score is 14, which indicates moderate depression."
This was the gap I kept hitting: the AI could build the data collection layer, but not the scoring layer.
The Reverse Item Problem
To test Claude's understanding of reverse scoring, I then asked it to build a WHO-5 Wellbeing Index. The WHO-5 has 5 items, all positively worded, scored 0-5. But here's the catch: higher scores mean better wellbeing. If you want to use it as a depression screen, you need to invert the scores (100 - raw score × 4 = percentage, with <50 indicating likely depression).
I told Claude:
"Build a WHO-5 Wellbeing Index. 5 items, 6-point scale (0=At no time to 5=All of the time). This is a positive-direction scale — higher scores mean better wellbeing."
Claude built the form correctly:
field_add --app <appId> --id w1 --type radio --title "I have felt cheerful and in good spirits" --options "At no time:0,Some of the time:1,Less than half:2,About half:3,More than half:4,All of the time:5"
All 5 items, correct scoring. Then I asked: "If someone scores all 5s, what's their wellbeing percentage?"
Claude said: "100%. Raw score 25, multiplied by 4 = 100."
Correct. Then I asked the real question: "How would you set up a reverse-scored version of this scale?"
Claude's answer: "I would negate the scores for reverse-scored items. For item 'I felt cheerful,' the options would be scored 5=At no time, 0=All of the time, reversing the direction."
Wait. Claude reversed the scores on the options — swapping 0 and 5 on every option. That's not reverse scoring. Reverse scoring means: for this specific item, invert the score after the respondent answers. The option labels stay the same; the score mapping flips.
Claude's approach would work if the item was negatively worded. But the WHO-5 items are all positively worded — you don't reverse-score a positively worded item. Claude conflated "reverse the option scores" with "reverse scoring as a psychometric technique."
Where the Confusion Comes From
The confusion is understandable. "Reverse scoring" is an ambiguous term:
- Textual reversal — rewording an item negatively ("I do NOT feel cheerful")
- Score reversal — inverting the numeric score for specific items (item score = max + min - raw score)
- Scale direction — whether higher total = better (positive direction) or higher total = worse (negative direction)
Claude mixed up all three. It thought "reverse scoring" meant swapping the option scores (approach 2), which would only make sense if the item was negatively worded (approach 1). And it didn't consider the scale's overall direction (approach 3).
The Fix: Explicit Instructions
I found that the key was being very explicit about what kind of "reverse" I meant. Instead of saying "add a reverse-scored item," I now say:
"Add item q9: 'I have thoughts that I would be better off dead.' This item is in the negative direction — higher scores indicate worse outcomes. Score the options 0=Not at all, 3=Nearly every day, same as other items. The reverse-scoring will be handled at the scale level, not the item level."
When I phrase it this way — explicitly saying "same as other items" and "handled at the scale level" — Claude stops trying to invert option scores and just builds the field normally.
The field_set_property Workaround
Since the MCP tools couldn't configure scale-level scoring at the time, I've started using field_set_property as a workaround for option-level score fixes.
If Claude builds a radio field with wrong scores, I can fix individual option scores without rebuilding the field:
# Fix the score on option index 2
field_set_property --app <appId> --id q3 --property options.2.score --value 2
This updates exactly one option's score. It's not scale-level scoring, but it lets me correct per-option scoring mistakes that Claude makes when it misunderstands the scoring direction.
The Real Lesson: Assessment Theory Is Hard
The deeper lesson here isn't about the MCP tools or the schema descriptions. It's that assessment theory is genuinely complicated, and LLMs have a surface-level understanding of it that breaks down on edge cases.
Claude knows:
- What a Likert scale is ✅
- What reverse scoring means (conceptually) ✅
- How to format
label:scoreoptions ✅ - When to use radio vs scale fields ✅
Claude doesn't reliably know:
- Which items should be reverse-scored vs negatively worded ❌
- The difference between item-level reversal and scale-level reversal ❌
- When higher scores should mean better vs worse (scale direction) ❌
- How to configure dimensional scoring (not exposed via MCP) ❌
This isn't a Claude-specific limitation. It's the gap between knowing a concept and applying it in a specific tool context. The same gap exists for any domain — the AI knows what a database index is, but it might create one on the wrong column.
What I'm Doing About It
I'm adding more context to the MCP tool descriptions. For field_add, the options parameter now reads:
options: z.string().optional().describe('Options list, comma-separated, supports label:score. For scored scales, use label:score format. Higher scores should align with the construct direction — if the scale measures depression, higher scores = more depression.'),
And for scoring-related fields:
score: z.number().optional().describe('Question score weight. Use for items that should contribute more/less to the total. Default is 1 (equal weight).'),
These descriptions won't make Claude a psychometrics expert. But they nudge it in the right direction — explicitly stating that "higher scores should align with the construct direction" prevents the most common mistake (inverting scores when the scale is already in the right direction).
The rest — dimensional grouping, cutoff scores, report templates — will come when I expose the Scale module through MCP. That's the next big piece of work. Until then, the AI can build the form, and I handle the scoring by hand.
The formlm-cli MCP server supports per-option scoring with label:score format. Open source at github.com/formlm/cli — try the platform at formlm.me.
Top comments (0)