Why AI Invents Sources: Inside Citation Hallucinations, Legal Risks, and How to Stop Them

Originally published on CoreProse KB-incidents

Large language models (LLMs) often produce confident citations to cases, papers, and URLs that do not exist. This is not a minor glitch; it follows directly from how they are built.

For lawyers, researchers, and product teams, knowing why AI fabricates references—and how to prevent it—is now essential to avoid sanctions, malpractice, and research misconduct.

This article explains:

• Why citation hallucination is baked into LLMs
• How it led to sanctions in Mata v. Avianca
• How to spot fake references before they cause damage
• Design patterns that reduce and govern the risk

---

1. From Prediction to Fabrication: Why LLMs Hallucinate Citations

LLMs are next-token predictors trained to produce likely text, not verified facts. They optimize for fluency, not truth or existence of sources.

When you ask for a citation, the model:

• Does not query a structured database
• Does not check case reporters, indexes, or URLs
• Simply continues a text pattern that “looks like” a citation

Citation formats as patterns, not knowledge

Citation styles (Bluebook, APA, MLA, OSCOLA, DOIs, URLs, ISBNs) are highly regular. Models learn these as patterns, so they can generate:

• Realistic case names and reporter formats
• Academic-sounding article titles
• Valid-looking DOIs and URLs

But:

• The structure can be correct
• The underlying source can be entirely fictional

Key idea

For an LLM, a “citation” is just plausible text, not a pointer into a verified catalog.

Human grounding vs. model interpolation

Humans:

• Read actual documents
• Cite specific, inspectable sources

LLMs:

• Predict plausible authors, titles, and venues
• Mix themes and keywords into convincing combinations
• Attach realistic years, volumes, and pages

Result: references that look credible but map to nothing.

Ghost sources are therefore:

• Not rare edge cases
• A predictable failure mode of systems rewarded for fluent projection, not verification

Data gaps, domain skew, and overconfidence

Citation hallucination worsens when:

• Domains are niche or underrepresented in training data
• The model has few concrete examples to draw from

Then the model:

• Interpolates between fragments
• Fills gaps with “best guess” references

Decoding and instruction-tuning often amplify this:

• Low temperature: pushes toward a single “most likely” (even if false) answer
• Helpfulness tuning: encourages:
  • Always giving an answer
  • Avoiding “I don’t know”
  • Presenting guesses confidently

Section takeaway

If you ask a text-only LLM to “provide citations,” you should assume some will be fake unless you explicitly design against it.

---

2. The Mata v. Avianca Warning Shot: When Fake Citations Reach the Courtroom

Mata v. Avianca is a high-profile example of ghost sources leading to sanctions. An attorney used ChatGPT to draft a federal court filing in the Southern District of New York. The brief contained multiple invented cases.

The model:

• Generated plausible case names, docket numbers, quotations, and reporter cites
• Produced opinions that looked authentic but did not exist

From technical hallucination to ethical violation

The core failure was not that the model hallucinated—that was foreseeable. The failure was that the attorney:

• Treated AI output as if it were real legal research
• Did not verify cases in Westlaw, Lexis, Bloomberg Law, or official reporters
• Filed the brief as a signed submission

When the court and opposing counsel could not locate the cases, the judge:

• Ordered the attorney to produce the opinions
• Received further AI-generated fake “full text” decisions
• Concluded the sources were fabricated

Case insight

The court held that using generative AI does not excuse Rule 11 obligations or basic due diligence.

Sanctions and emerging norms

The court imposed:

• Monetary sanctions
• Mandatory notification to the lawyer’s firm and affected clients
• Strong criticism of the lawyers’ conduct

Key principles from the opinion and emerging guidance:

• Lawyers remain fully responsible for AI-assisted filings
• It is foreseeable that LLMs hallucinate citations
• Failing to verify AI-generated references can itself be sanctionable

Courts and bar associations increasingly require:

• Disclosure if generative AI was used
• Certification that AI-generated content has been independently checked

Parallel risks in other domains

The same pattern appears in other regulated and research-heavy fields:

• Medicine

  • Fake citations to non-existent clinical trials
  • References that appear to support unproven therapies

• Finance

  • Invented analyst reports or white papers
  • Fabricated citations to regulatory guidance

• Academia and research

  • AI-generated bibliographies with non-existent articles
  • Students or researchers citing hallucinated sources, triggering misconduct concerns

Section takeaway

After Mata, organizations cannot credibly claim surprise when AI hallucinations lead to regulatory or professional consequences.

---

3. Recognizing Ghost Sources: Patterns, Red Flags, and Detection Tactics

To manage risk, you need a practical way to spot fabricated citations before they reach courts, regulators, or publication.

The same structural behaviors that cause hallucinations leave detectable fingerprints.

Common patterns of fake citations

Ghost sources often show:

• Unfamiliar venues with perfect formatting

  • Journals or law reviews with realistic names
  • No presence in major indexes (PubMed, Web of Science, HeinOnline, SSRN, etc.)

• Recombined case names

  • Real-sounding party pairings that do not exist in any reporter
  • Docket numbers that do not match court conventions

• Too-neat metadata

  • DOIs/URLs with correct syntax but:
  • Return 404
  • Resolve to unrelated content
  • Have no matching article on the publisher site

• Generic, “on trend” titles

  • Vague but topical titles that vanish in database searches

Quick pattern check

If a citation looks perfect but cannot be found in authoritative databases, treat it as suspect.

Practical red flags for reviewers

When reviewing AI-generated references, watch for:

• Incomplete or inconsistent details

  • Missing volume/issue/page numbers
  • Mixed citation styles in one reference

• Suspicious recency or precision

  • Very recent years in slow-to-index journals
  • Repeatedly tidy page ranges (e.g., 45–47, 101–103)

• Non-resolving legal citations

  • Cases absent from Westlaw, Lexis, Bloomberg Law, or official reporters
  • Docket formats inconsistent with the named court

For web sources:

• Mismatched domains (e.g., “official guidance” on a random blog)
• URLs whose content does not match the described title or author

These checks are fast and can be standardized.

Automated and embedding-based detection

Product teams can build automated defenses so humans are not the only safeguard:

• URL resolution checks

  • Resolve every URL and log:
  • Status codes
  • Redirect chains
  • Basic content fingerprints
  • Flag 404s, unexpected redirects, or mismatched content

• Cross-referencing against indexes

  • Academic: PubMed, Crossref, arXiv, Web of Science, Scopus, Google Scholar
  • Law: Westlaw, Lexis, Bloomberg Law, court APIs
  • Internal: document management systems, knowledge graphs

• Embedding-based verification

  • Index trusted corpora with embeddings
  • For each claimed citation, search for close semantic matches
  • Flag references with no close match as likely hallucinations

You can also run a “verification pass”:

• A second model or service checks each citation against search or retrieval results
• Labels each as:
  • Verified
  • Ambiguous / partial match
  • No match (likely hallucination)

Section takeaway

A small set of automated checks plus targeted human review can catch most ghost sources—if you design for verification explicitly.

---

4. Designing Against Hallucinations: Grounding, RAG, and Safer Citation Flows

The most effective control is architectural: constrain what the model can cite and verify what it outputs. Do not rely on “be honest” instructions alone.

Grounding and retrieval-augmented generation (RAG)

Grounding and RAG restrict the model to a curated corpus. Instead of inventing citations, the model selects and summarizes from known documents.

Typical workflow:

1. Retrieve relevant documents from verified indexes based on the user’s query.
2. Pass those documents (or excerpts) to the model as context.
3. Instruct the model to:
   • Cite only from the provided set
   • Use explicit identifiers (case IDs, DOIs, document IDs)

This converts the task from free-form invention to constrained summarization.

Design pattern

“You may only cite documents from this list, using their IDs, and must not invent new sources” should be enforced in prompts and in code.

You can further:

• Post-process outputs to:
  • Map internal IDs to full citations
  • Reject any reference not tied to a known ID
• Log all cited IDs for audit and re-checking

Combined with verification steps from Section 3, this significantly reduces hallucinated citations.

---

Conclusion

Citation hallucinations are a structural feature of LLMs, not a rare bug. They arise because models:

• Predict plausible text, not verified facts
• Treat citation formats as patterns, not as pointers to real documents

In law, Mata v. Avianca showed how this can escalate into sanctions when lawyers rely on AI without verification. Similar risks exist in medicine, finance, and research.

To manage these risks:

• Assume ungrounded models will fabricate some references
• Train reviewers to recognize common patterns and red flags
• Build automated checks against trusted indexes and URLs
• Use grounding and RAG so models can only cite from verified corpora

With these controls, organizations can use generative AI productively while keeping ghost sources out of courts, reports, and publications.

---

About CoreProse: Research-first AI content generation with verified citations. Zero hallucinations.

🔗 Try CoreProse | 📚 More KB Incidents