LLMs Misread Historical Ciphers — A Verification Workflow

You see a headline claiming an LLM solved a World‑War‑II Enigma message that has resisted decryption since 2005. The excitement is real, but the model’s output can look plausible while still being wrong. Before you trust the result, you need a way to check it.

What you'll learn:

• How LLMs can produce fluent but incorrect cipher solutions
• A lightweight Python script that scores candidate plaintexts
• Three verification strategies and when each is appropriate

Why LLMs Can Appear Correct Even When Wrong

Language models generate text by predicting the next token, not by reasoning about the underlying cryptographic structure. When faced with a short ciphertext they may produce a plaintext that follows English letter frequencies and contains common words, yet the mapping does not match the actual encryption key. This fluency creates a false sense of correctness that can fool a quick glance.

A Simple Validation Script

The following function takes a candidate plaintext and returns a confidence score based on two heuristics: chi‑squared distance from typical English letter frequencies and the proportion of words found in a small dictionary. Higher scores indicate closer resemblance to natural English.

import string
from collections import Counter
import math

ENGLISH_FREQ = {
    'a': .08167, 'b': .01492, 'c': .02782, 'd': .04253,
    'e': .12702, 'f': .02228, 'g': .02015, 'h': .06094,
    'i': .06966, 'j': .00153, 'k': .00772, 'l': .04025,
    'm': .02406, 'n': .06749, 'o': .07507, 'p': .01929,
    'q': .00095, 'r': .05987, 's': .06327, 't': .09056,
    'u': .02758, 'v': .00978, 'w': .02360, 'x': .00150,
    'y': .01974, 'z': .00074
}

SIMPLE_DICTIONARY = {
    'the', 'and', 'for', 'are', 'but', 'not', 'you', 'all',
    'can', 'had', 'her', 'was', 'one', 'our', 'out', 'day',
    'get', 'has', 'him', 'his', 'how', 'its', 'may', 'new',
    'now', 'old', 'see', 'two', 'who', 'boy', 'did', 'man',
    'men', 'put', 'too', 'use'
}

def score_plaintext(text: str) -> float:
    """Return a higher score for text that looks like English."""
    # keep only letters
    letters = [c.lower() for c in text if c.isalpha()]
    if not letters:
        return 0.0
    total = len(letters)
    observed = Counter(letters)
    chi = sum(((observed.get(ch, 0) / total - ENGLISH_FREQ[ch]) ** 2) / ENGLISH_FREQ[ch]
              for ch in ENGLISH_FREQ)
    # chi‑squared lower is better; invert for scoring
    freq_score = 1 / (1 + chi)
    # word proportion
    words = ''.join(c if c.isalpha() else ' ' for c in text).lower().split()
    if not words:
        word_score = 0.0
    else:
        matches = sum(1 for w in words if w in SIMPLE_DICTIONARY)
        word_score = matches / len(words)
    return 0.6 * freq_score + 0.4 * word_score

Example usage

If you save the function above as scorer.py, you can test a single candidate like this:

if __name__ == '__main__':
    candidate = "HELLO WORLD THIS IS A TEST"
    print(f"Score: {score_plaintext(candidate):.3f}")

Why this works: The chi‑squared component penalizes unusual letter distributions, while the word‑proportion component rewards recognizable vocabulary. Together they catch many hallucinated outputs that are fluent but statistically odd.

Running the scorer from the command line

You can invoke the scorer directly from a terminal to test many candidates quickly. Save the function in a file named scorer.py and use this tiny wrapper:

#!/usr/bin/env python3
import sys
from scorer import score_plaintext

def main():
    for line in sys.stdin:
        text = line.strip()
        if text:
            print(f"{text}\t{score_plaintext(text):.3f}")

if __name__ == "__main__":
    main()

Make it executable (chmod +x score.py) and run:

./score.py < candidates.txt

This prints each candidate followed by its score, letting you spot low‑scoring hallucinations at a glance.

Three Verification Strategies

Approach	Tradeoff	When to Use
Manual expert review	High accuracy, slow, needs domain knowledge	Final validation of high‑value claims
Automated statistical checks	Fast, objective, may miss subtle errors	Early screening of many candidates
Cross‑model consensus	Reduces model‑specific bias, still can share blind spots	When you have access to multiple LLMs

Pick the method that matches your timeline and the cost of a mistake.

Common Failure Modes

• Over‑reliance on fluency: A model may produce grammatical text that ignores the cipher’s constraints.
• Ignoring historical context: Assuming a modern word list applies to wartime communications can lead to false positives.
• Assuming uniqueness: Multiple plausible plaintexts can score similarly; the model picks one arbitrarily.
• Misinterpreting nulls: Treating padding or operator errors as meaningful letters distorts the frequency analysis.

Being aware of these pitfalls helps you decide when a candidate needs deeper scrutiny.

Putting It All Together: Workflow

1. Obtain the LLM’s proposed plaintext for the ciphertext.
2. Run score_plaintext on the candidate; if the score is below a chosen threshold (e.g., 0.45), treat it as suspect.
3. If the score is borderline, try alternative prompts or temperature settings to generate additional candidates.
4. Apply a second verification method: either run a different LLM and compare scores, or consult a subject‑matter expert for a manual check.
5. Only accept the solution when at least two independent checks agree and the statistical score indicates English‑like patterns.

Key Takeaways

• LLM fluency does not guarantee cryptographic correctness.
• A simple statistical‑word‑based scorer can flag many hallucinations quickly.
• Combine automated checks with expert review or model consensus for high‑stakes claims.
• Watch for context‑specific failures like anachronistic vocabulary or multiple plausible solutions.
• Document your verification steps so others can reproduce the validation.

Source

OpenAI GPT–6 Astra breaks Enigma message that has resisted solution since 2005
I added a practical verification workflow, a working Python scoring script, and a comparison of validation strategies that the source does not cover.

Support this work

These write-ups are researched and published with no paywall, sponsor, or tracking. If one saved you an afternoon, a small tip keeps them coming.

USDT, USDC or USDD · TRC-20 (Tron)

TFTNsfyomKrnUutRjBTGVULp19ByW29KbY