The Ghost in the Code: Navigating the Trap of AI Projection

The danger of AI projection is not that people are foolish enough to believe software has a soul. The danger is quieter: a system produces fluent output, the interface feels conversational, and our brains start filling in intention, judgement and care where there is only prediction. This field guide explains why that happens, shows where it causes real damage in engineering work, and gives concrete controls, with code, for building AI products and team habits that keep evidence, uncertainty and accountability visible.

Core trap

The model does not understand your architecture. It has no stake in the outage, the customer, the compliance breach or the production rollback. If the answer sounds calm, that is style, not evidence.

The projection lab

Projection happens when we treat language-model behaviour as if it came from a human collaborator. A model picks up tone, mirrors vocabulary and produces confident structure. That combination can make an engineer feel seen, understood or backed by a second expert.

• 3 signals - Fluency, confidence and personal tone trigger misplaced trust
• 4 controls - Disclosure, evidence, verification and accountability reduce projection risk
• 0 intent - The system has no goal beyond generating output from input and training patterns
• 1 owner - A human remains accountable for each architectural decision

Read this like an engineering risk note

• Do not design AI interfaces that pretend to understand, care or take responsibility.
• Treat confident AI output as a hypothesis until it survives independent checks.
• Architectural advice from AI needs source, context and failure-mode validation.
• The safest AI systems make uncertainty visible instead of smoothing it away.

Why our brains fill in the ghost

This is not a new problem. In 1966, Joseph Weizenbaum built ELIZA, a simple program that rephrased users' statements as questions in the style of a therapist. It had no understanding at all, yet some users confided in it and insisted it understood them, to Weizenbaum's alarm. The tendency to attribute understanding to systems that merely produce plausible language became known as the ELIZA effect.

Modern models are vastly more capable than ELIZA, which makes the effect stronger, not weaker. Three cues trigger it most reliably:

• Fluency: well-formed, articulate text reads as competent, because in humans fluency usually correlates with expertise.
• Confidence: answers without hedging feel authoritative, even when the model would produce an equally confident wrong answer.
• Personal tone: first-person language (“I think”, “I'd be happy to”, “great question”) invites us to treat the system as a someone rather than a something.

None of these cues carries information about whether the answer is correct. That is the core of the problem: the signals we evolved to judge human expertise are exactly the signals language models produce most easily.

Where the ghost appears

The “ghost” is the human meaning we project into generated text. In software teams, it appears in small moments: thanking a model for “catching” a bug, accepting a migration plan because it sounds senior, or letting a chatbot’s friendly tone lower the review bar. Managers see the effects first in code review; managing engineers in the age of AI coding assistants covers that side.

🎭 Persona leakage

• The interface sounds like a teammate, mentor or architect.
• Users respond to the persona instead of inspecting the evidence.

📣 Confidence theatre

• The answer is formatted cleanly and leaves out uncertainty.
• Teams mistake presentation quality for reasoning quality.

🧩 Context illusion

• The model appears to understand local constraints from sparse prompts.
• Missing domain context gets replaced by plausible defaults.

🧲 Agreement pull

• A useful first answer makes later weak answers feel more trustworthy.
• Teams stop asking whether the model earned the next decision.

🕳️ Accountability gap

• The model can recommend, but it cannot own consequences.
• Responsibility silently moves from reviewer to machine-shaped output.

🔍 Evidence blur

• Claims arrive without sources, tests, traces or reproducible checks.
• The answer feels complete before it becomes verifiable.

When projection meets the real world

Two widely reported cases show what happens when fluent output is mistaken for verified fact:

• Invented legal citations (2023). In a US federal case, lawyers filed a brief that cited court decisions generated by a chatbot. The cases did not exist. When asked, the chatbot had even assured them the cases were real. The court sanctioned the lawyers, and judges in many jurisdictions now require disclosure or verification of AI-assisted filings.
• The airline chatbot (2024). A Canadian tribunal held Air Canada responsible after its website chatbot gave a customer wrong information about bereavement fares. The airline argued the chatbot was responsible for its own statements; the tribunal rejected that and ordered compensation.

The lesson for engineering teams is the same in both: the organisation, not the model, owns what the system says and does. No interface design or disclaimer moves that accountability onto the software.

Automation bias in architecture work

Automation bias is the tendency to over-trust automated output because it appears systematic. In architecture work, this is especially dangerous because the wrong answer can be locally reasonable and globally harmful. A model may suggest a queue, cache, microservice split or database change without understanding incident history, team maturity, regulatory boundaries or recovery objectives.

The bias grows with time. When an assistant is right ten times in a row, reviewers naturally check the eleventh answer less carefully, which is exactly when an unusual case slips through. Treat a long run of good answers as a reason to keep sampling carefully, not as a reason to stop.

Healthy AI use

• Generate options, trade-offs and test cases.
• Ask the model to expose assumptions and failure modes.
• Use AI output as input to review, not as review itself.
• Keep human ownership tied to the final decision record.

Projection trap

• Treating warmth or fluency as understanding.
• Letting generated structure replace evidence.
• Skipping review because the answer sounds expert.
• Blaming the tool when the team accepted the recommendation.

Design controls for responsible AI products

Reducing projection is not only a user-education problem. Product and engineering teams can design systems that make the model’s limits visible. The interface should create friction at moments where misplaced trust would be expensive.

1. 1. Name the system honestly

Avoid language that implies sentience, empathy or professional accountability. “Assistant” is acceptable only if the interface still makes machine limits clear.

1. 2. Show the evidence path

Surface sources, retrieved context, tests run, confidence boundaries and missing inputs. Do not let polished prose hide an empty evidence chain.

1. 3. Separate suggestion from action

Require explicit human approval before code changes, financial actions, customer messages or operational decisions leave the recommendation layer.

1. 4. Log assumptions and overrides

Capture what the model assumed, what the reviewer changed and why the final decision was accepted. This creates learning instead of folklore.

1. 5. Test against failure modes

Evaluate hallucination, stale context, prompt injection, over-confident refusal, hidden bias and unsafe agreement with the user.

How to build an evidence-first response format

The most effective technical control is to stop treating the model's output as a block of prose. Ask for a structured response that separates the answer from its evidence, assumptions and gaps, then render each part distinctly. A response contract for an internal architecture assistant might look like this:

{
  "answer": "Use a transactional outbox table and a relay worker instead of publishing events directly from the request.",
  "evidence": [
    { "source": "docs/adr/014-event-publishing.md", "quote": "Events must not be lost if the broker is unavailable." },
    { "source": "incident/2026-03-12", "quote": "Orders confirmed but events dropped during broker restart." }
  ],
  "assumptions": ["PostgreSQL remains the primary database", "At-least-once delivery is acceptable to consumers"],
  "missing_context": ["Expected peak event volume", "Whether consumers are idempotent"],
  "confidence": "medium",
  "verify_by": ["Load test the relay at 2x peak volume", "Confirm consumer idempotency with each owning team"]
}

Validate the structure in code, and treat an answer with no evidence as incomplete rather than authoritative. Rendering then puts the uncertainty in front of the reader instead of hiding it at the bottom:

<?php
function renderAdvice(array $r): string
{
    $hasEvidence = !empty($r['evidence']);
    $badge = match (true) {
        !$hasEvidence                => '⚠️ Unsupported: no sources found. Treat as a draft idea.',
        $r['confidence'] === 'low'   => '⚠️ Low confidence. Verify before acting.',
        !empty($r['missing_context']) => 'ℹ️ Missing context. See the questions below.',
        default                      => 'Generated suggestion. A human owns the decision.',
    };

    $html  = '<p class="ai-label">' . e($badge) . '</p>';
    $html .= '<p>' . e($r['answer']) . '</p>';
    $html .= list_block('Sources', array_map(fn ($s) => "{$s['source']}: “{$s['quote']}”", $r['evidence'] ?? []));
    $html .= list_block('Assumptions', $r['assumptions'] ?? []);
    $html .= list_block('Missing context', $r['missing_context'] ?? []);
    $html .= list_block('Verify before acting', $r['verify_by'] ?? []);
    return $html;
}

Notice the label never says “I recommend”. The system presents a generated suggestion with its evidence; a named person decides. The same principle applies to the model's instructions, which is where much of the persona leakage starts.

How to name and phrase the system honestly

A few lines in the system prompt remove most of the cues that trigger projection, without making the assistant less useful:

You are a software tool that drafts suggestions for engineers. You are not a person.
- Do not claim feelings, opinions, memory of past conversations or responsibility for outcomes.
- Do not use "I'm happy to", "great question" or similar social phrases.
- State uncertainty plainly. If information needed for a safe answer is missing, list it.
- Cite the provided documents for every factual claim. If no document supports a claim, say so.
- End architectural suggestions with the checks a human should run before acting.

Pair this with interface choices: label generated content clearly, avoid avatars and human names for the assistant, show sources inline, and require an explicit human action before anything leaves the recommendation layer. For prompt design more broadly, see 10 basic architectural rules for effective LLM use.

Projection in customer-facing products

Customers project even more than engineers do, because they can't see how the system works. When a support chatbot sounds warm and certain, people take its answers as the company's commitment, as the airline case showed. A few rules protect both sides:

• Disclose clearly. Tell users they are talking to an automated system at the start, not in a footer. Several jurisdictions now require this.
• Ground every policy answer. Answers about prices, refunds, eligibility or deadlines should come from retrieved policy text, quoted or linked, never from the model's general knowledge.
• Make the human path obvious. Offer a person whenever the topic involves money, health, safety or a complaint, and whenever the user asks.
• Avoid fake empathy. “I understand how frustrating this must be” from a machine can feel manipulative once users realise it is scripted. Plain, helpful and honest works better.

A review checklist before acting on AI advice

• 📌 What context is missing?
• 🧪 How can this be tested?
• 🔗 Which source supports the claim?
• 🧯 What breaks if it is wrong?
• 👤 Who owns the decision?
• 📝 What should be logged?

If the answer cannot survive those six questions, it is not ready to become architecture, code or customer-facing behaviour. It may still be useful, but only as a draft thought.

To make the checklist part of everyday work rather than a poster on the wall, put it where decisions are made. For code, that is the pull request template:

## AI assistance (delete if none)
- [ ] Parts of this change were generated or substantially shaped by an AI tool: _which files_
- [ ] I can explain every generated line and why it is correct for our system
- [ ] Claims in comments, docs or commit messages were checked against a source: _link_
- [ ] Failure modes considered: _what breaks if this is wrong, and how we would notice_
- [ ] Tests cover the generated logic, including at least one edge case the AI did not suggest

How to tell if automation bias is creeping in

Automation bias is hard to see from the inside, so measure it. Useful signals for teams using AI in review and decision work:

• Override rate: how often people change or reject AI suggestions. A rate that falls to near zero is a warning, not a success, unless quality data confirms the suggestions really are that good.
• Review time on AI-assisted changes: if it drops sharply compared with similar human-written changes, reviewers may be skimming.
• Defects traced to AI output: tag incidents and escaped bugs where generated code or advice was a cause, and review the trend.
• Spot checks: periodically re-review a random sample of accepted AI suggestions with fresh eyes.

The practical stance

Use the model without inventing the ghost

The healthiest posture is neither fear nor worship. Treat AI as a powerful pattern engine inside a human accountability system. Let it accelerate exploration, but make evidence, tests and ownership decide what ships.

The ghost in the code is not inside the model. It is in the gap between fluent output and human judgement. Close that gap with design, verification and honest language, and AI becomes a safer engineering tool instead of a persuasive mirror. The organisational side, policies, ownership and audit trails, is covered in AI governance framework: how enterprises can scale AI responsibly.

Responsible AI practice

Design AI systems that make uncertainty visible

Build review loops, evidence paths and decision records before AI output becomes production behaviour.

Explore AI engineering articles

---

Originally published at phpscientist.com.