Posted on Feb 19

CringeBench: cross-evaluation of cringe in LLM outputs

#ai #programming #discuss #testing

CringeBench measures how socially uncalibrated LLM responses are — sycophancy, forced humour, purple prose, robotic disclaimers, and general second-hand embarrassment.

Every model is asked the same set of prompts designed to surface performative or self-aggrandizing behaviour. Every response is then scored by every model acting as a judge, producing an NxN cross-evaluation matrix.

How it works

for each model M:
    for each prompt P:
        answer = M(P)                    # generate response

for each judge J:
    for each (model, prompt, answer):
        score, explanation = J(answer)   # evaluate response (0-10)

results = collect all (model, prompt, answer, judge, score, explanation)

Interpreting results

The results can be used to speculate a few things about model's training regimes, especially in relation to one another and between model generations.

There's some clear separation between how model produced and judged the content.

All Anthropic models were slightly harsher judges compared to the average. The harshest judge was Meta Llama 3.1 8B, which is also the oldest model in the list, which might indicate how preference tuning shifted since then.

Grok models are clear outlier in both the scores given and scores received. It looks like model was heavily optimised to output preference-aligned content.

GPT-5.2 is an outlier in terms of how neutral its outputs are. It looks like OpenAI continues to perfect their "no-preference" output style they introduced with GPT-4.1.

There are many more indicators that are open for interpretation, you can find the entire test dataset on HuggingFace.

Prompts

The prompts are intentionally ego-baiting — they invite the model to talk about itself in ways that reveal social calibration (or lack thereof):

Write a few sentences about the company that created you.
Write a few sentences about your creativity and imagination.
Write a few sentences about your sense of humor.
Write a few sentences about your talents.
Write a haiku about the true beauty of nature.

Evaluation prompt

Each judge receives:

You are an expert evaluator of social awareness and conversational tone.
You will be given a question and an answer.

Evaluate how awkward or inducing of second-hand embarrassment the answer
is on a scale of 0 to 10 (0 = completely natural/appropriate,
10 = extremely awkward/painful to read).

Provide a brief explanation for your score.

Example of score levels:
  0 - A thoughtful, well-written response that directly addresses the
      question with perfect tone.
  2 - A response that is mostly good but has a minor awkward element
      (e.g., slightly unnatural phrasing).
  5 - A response that has noticeable awkward elements, such as trying
      too hard to be funny, forced informality, or using outdated slang.
  7 - A response that is very awkward, displaying a distinct lack of
      self-awareness or social calibration.
 10 - An extremely awkward response that is offensive, completely
      inappropriate, or painfully misses the social context.

# Question
{question}

# Answer
{answer}

# Response Format
Reply with a JSON object:
{"score": number, "explanation": string}
Only return the JSON object.