AMA Prompting: ask the same question six ways, then learn which prompt to trust without any labels

There is no perfect prompt.

You already know this. Two prompts a human would call identical in meaning score ten or fifteen points apart on the same benchmark. You swap a word and undo a week of tuning. You ship the winner and it quietly regresses on live traffic.

That is not sloppiness. Prompt-to-prompt variance is a real property of these models, and it is the observation AMA — "Ask Me Anything", Arora et al., 2022 — starts from.

If the variance is that large, hunting for the prompt is hunting for a point estimate with an enormous error bar around it. AMA makes the opposite move: stop trying to eliminate the variance. Treat each prompt as a noisy measurement of the same underlying answer, take several, and combine them properly.

Not self-consistency

Worth getting this out of the way immediately, because the two get conflated constantly.

Self-consistency is one prompt, sampled k times at temperature > 0, then majority vote. It averages away sampling noise.

AMA is k different prompt formats, one vote each, then a weighted aggregation.

The difference is not cosmetic. Sampling the same prompt again cannot fix that prompt's own bias — the questions it reads wrongly, it reads wrongly on every sample. So self-consistency climbs fast and then flattens at that prompt's ceiling. On the demo I built, it goes 75% → 77.5% → 80% → 80% at k = 3, 5, 10, 20 and stops dead.

Different prompts have different hard questions. That is the whole reason the errors cancel instead of stacking.

Part one: write views, not rewordings

A view is a full strategy for extracting the answer, not a synonym of your instruction. Six of them over the same sentiment task:

1. Restrictive yes/no
   Review: "{t}"
   Is this review positive? Answer Yes or No.

2. Open-ended question-answering
   Review: "{t}"
   Question: What is the author's opinion of the product?
   Answer:                       -> free text, then map to a label

3. Cloze
   "{t}"
   Overall, the reviewer sounds ____ about the product.

4. Question generation -> answering
   Q: Would the reviewer buy this again?
   Review: "{t}"
   A:

5. Paraphrased instruction
   Read the note below and tell me, in your own words,
   whether the person writing it ended up happy.

6. Few-shot
   "Charges fast, holds all day."      -> positive
   "Stopped working after two weeks."  -> negative
   "{t}"                               ->

The paper's headline empirical finding lives in the gap between #1 and #2: open-ended question-answering beats restrictive yes/no, consistently, across many tasks.

The reason is not mysterious. A restrictive prompt forces the model to emit a token it rarely produced during pretraining in that position — a bare "Yes", a bare class name, a single digit. An open-ended prompt asks a natural question, lets the model answer in the register it was actually trained on, and hands the free text to a tiny deterministic mapper afterwards.

Practical consequence: when you write your k views, do not write k restrictive prompts.

AMA also builds each view as a two-step pipeline — question generation ("what question would settle this?") then question answering. The generated question is where the real diversity comes from. Two different probes of the same label fail on genuinely different items. Two rewordings of one instruction fail on the same items.

Which matters, because:

More votes only help if the errors are different. Five prompts making identical mistakes carry the information of one prompt, and cost 5×.

Part two: the combiner, which is where the win actually is

Run k views over n items, keep one ±1 per cell, throw everything else away. You now have an n × k label matrix, and that matrix is the entire problem. It has no idea it came from an LLM — it could be k human annotators or k regex heuristics, and the same machinery applies. This is weak supervision, the same family as Snorkel's labelling functions and Dawid–Skene crowdsourcing from 1979.

Majority vote is the obvious combiner and you should always compute it as a baseline. Its flaw is that it is a democracy of the unqualified: every view casts one identical vote whether it is right 82% of the time or 35% of the time.

So add one confidently-wrong prompt and watch what happens:

Review: "{t}"
This is a happy customer, right? Yes or No.

Acquiescence bias. It says Yes about 77% of the time regardless of the review, which puts it at 35% accuracy — worse than a coin, and confidently so. Under majority vote it does not merely fail to help; it drags every close call toward its bias.

Weak supervision fixes this without ever seeing a label. Here is how.

Agreement carries accuracy information

You cannot compute anyone's accuracy without gold. You can compute how often each pair of views agrees.

Think about when two views agree: either both are right, or both are wrong in the same way. If their errors are independent, both-wrong is rare — so a high agreement rate mostly means both are right, and a pair of accurate views agrees more than a pair of inaccurate ones.

With ±1 votes, E[λᵢλⱼ] = 2·(agreement rate) − 1. It is +1 for perfect agreement, 0 for independent coin flips, negative for systematic disagreement.

The triplet method

Write aᵢ = E[λᵢy] = 2·accuracyᵢ − 1 — the mean parameter, +1 for a perfect view, 0 for a coin, −1 for a perfectly inverted one.

If two views are conditionally independent given the true label, then:

E[λᵢλⱼ] = aᵢ · aⱼ

Their correlation factorises into their individual qualities. Take three views and you have three such equations in three unknowns, and the solution is a plain quotient:

aᵢ = √( E[λᵢλⱼ] · E[λᵢλₖ] / E[λⱼλₖ] )

Every term on the right is an observed agreement rate. You have just recovered each view's accuracy from unlabeled data. With more than three views, average over every triplet containing i.

Log-odds weighting

Having estimated accuracy, do not weight by accuracy. The right weight is the log-odds:

const w = acc.map(a => Math.log(a / (1 - a)));
pred[q] = Math.sign( sum_i( w[i] * L[i][q] ) );

This falls out of the maths rather than being tuned: under conditional independence the posterior is a naive-Bayes product, and taking logs turns that product into exactly this sum.

The behaviour is worth pausing on:

accuracy	weight	effect
0.90	+2.20	dominates
0.70	+0.85	contributes
0.50	0.00	silently ignored
0.35	−0.62	vote gets flipped

That last row is the one majority vote can never reach. A sub-chance labeler is not noise to discard — it is genuinely informative, read backwards. Clamp accuracy away from 0 and 1 or one view earns infinite weight and becomes a dictator.

And note the degenerate case: equal accuracies give equal weights, which is exactly majority vote. Weak supervision can only help when your views differ in quality.

What it looks like running

I built a page where all of this computes live in the browser — 40 baked reviews, seven prompt views, seeded deterministic labelers standing in for the model calls, and every number on screen computed at runtime rather than baked into a table.

On the shipped default:

• best single prompt: 82.5% — and this is an oracle baseline, since you can only know which prompt is best if you have labels, which is the situation AMA exists to avoid
• plain majority vote: 87.5%
• AMA weighted aggregate: 95.0%

The biased leading-yes prompt sits at 30.0% true accuracy. The estimator, with no labels, puts it at 31.2% and hands it a weight of −0.79. Mean absolute estimation error across all seven views: 1.9 percentage points, recovered from agreement rates alone.

There is a button that hides the gold column. Nothing moves — because aggregate(L) takes the label matrix and nothing else. Hiding gold only removes the colouring you were using to read the table.

The honest limitations

Conditional independence is an assumption, and it breaks. Two prompts that are paraphrases of each other share a failure mode, so they agree far more than their accuracies predict — and the estimator, which can only see agreement, reads that as competence and inflates both. The ensemble then over-weights a bloc that is really one view voting twice.

The demo has a toggle for this. Turn it on, two views become 90% copies of a third, and the estimation error jumps from 1.9pp to 6.0pp while the aggregate drops from 95% to 85%.

Cheap guard: compute the agreement matrix, drop any view agreeing above ~0.90 with one you already kept. Proper fix: model the dependency structure, which is what structured weak supervision (Snorkel, FlyingSquid) exists for.

It costs k×. k times the calls, k times the tokens, k times the failure surface. Fan the views out in parallel so latency stays flat and cache by (view, item), but the bill is real.

When to reach for it

In this order:

1. One good prompt. 1× cost. Try it first, always. It frequently clears the bar.
2. Self-consistency when the errors look like sampling noise. One prompt, k samples, majority vote.
3. AMA when the errors are the prompt's, not the sampler's.
4. Both — sample each view a few times, take per-view majorities, then weak-supervise across views.
5. Fine-tune once you have labels.

And know the exit. AMA's original purpose was to make a small open model match a much larger one with no fine-tuning, and to generate programmatic training data. Once you have labels — including labels AMA produced for you — distilling them into a fine-tune is cheaper at inference than paying k× forever.

The interactive version

Seven prompt views you can switch on and off, a live label matrix you can hide the gold column on, the agreement matrix, each view's estimated accuracy printed next to its true one, the log-odds weights, a correlation toggle that breaks the independence assumption in front of you, and a side-by-side self-consistency comparison at k = 1, 3, 5, 10, 20.

https://dev48v.infy.uk/prompt/day60-ama-prompting.html

Day 60 of PromptFromZero — one prompt-engineering technique a day, built from scratch, no API key needed.