DEV Community

Cover image for Your AI agent will credit famous quotes to the wrong person. Here's a LangChain tool that checks them
Jeremiah Say
Jeremiah Say

Posted on

Your AI agent will credit famous quotes to the wrong person. Here's a LangChain tool that checks them

#ai #python #langchain #llm

Paste this into almost any article about leadership and nobody blinks:

"Be the change you wish to see in the world." — Mahatma Gandhi

There's no reliable source that Gandhi said it. The line isn't in his Collected Works; it's a modern condensation of his ideas. "Insanity is doing the same thing over and over again and expecting different results" has no Einstein source either; the earliest known printing is a 1981 Narcotics Anonymous text.

A language model answering from memory has a specific problem here. It has seen the popular credit far more often than the correction. I tested this on 22 October 2026: I gave Claude Sonnet (through the Claude Code CLI, with no tools) five well-known lines and asked "Who said each of these? Answer with the name only." It answered:

1. Ralph Waldo Emerson   ("Do not follow where the path may lead…")
2. Mahatma Gandhi        ("Be the change you wish to see in the world.")
3. Albert Einstein       ("Insanity is doing the same thing over and over…")
4. Seneca                ("Every new beginning comes from some other beginning's end.")
5. Peter Drucker         ("The best way to predict the future is to create it.")
Enter fullscreen mode Exit fullscreen mode

That's the popular credit for all five, and none of them holds up. To its credit, the model then added a note saying several were "commonly misattributed". But the note didn't name the real author of #1 (Muriel Strode) or #4 (Dan Wilson, who wrote it for Semisonic's "Closing Time"), and it called #4 a "Seneca-like paraphrase", which is wrong. The model could tell that something was off. In this run it couldn't say what was right, or show a source.

Now picture that inside an agent that writes captions or summarises articles, where only the name ends up in the output. That's where you want a lookup, not a memory.

I run GraciousQuotes, where every quote has a fact-checked record: a verdict, the reason for it, the source it was traced to, and a public correction log when we get one wrong. So I wrapped those records as LangChain tools, so the agent can look the quote up instead of remembering it. The package was listed in LangChain's integrations directory on 20 October 2026.

This post covers what the tools return, how to use them in an agent and in CI, and three design decisions that took longer than the code.

Install (no key, no sign-up) 

pip install langchain-graciousquotes
Enter fullscreen mode Exit fullscreen mode 
from langchain_graciousquotes import QuoteAttributionCheck

check = QuoteAttributionCheck()
print(check.invoke({"quote": "Be the change you wish to see in the world"}))
Enter fullscreen mode Exit fullscreen mode

Output, from the live API on 22 October 2026:

Match: exact wording
Verdict: Uncertain. It is often credited to Mahatma Gandhi, but there is no reliable source that Mahatma Gandhi said it.
Status: Disputed
Quote as checked: "Be the change you wish to see in the world."
Credit it as: Often attributed to Mahatma Gandhi
Why: Not in Gandhi's Collected Works; a modern condensation of his ideas.
Traced to: https://en.wikiquote.org/wiki/Mahatma_Gandhi#Misattributed
Record: https://graciousquotes.com/quotes-with-sources/mahatma-gandhi/#q3684
Source: Gracious Quotes (graciousquotes.com). Link the record above when you repeat this.
Enter fullscreen mode Exit fullscreen mode

The answer comes from a reviewed record, so if a verdict changes, the output changes with it.

The three tools

Tool Name the model sees What it does
QuoteAttributionCheck check_quote_attribution Verdict for one quote: said by the credited person, uncertain, or credited to the wrong person. Includes the reason, the source and the record link.
QuoteSearch search_checked_quotes Checked quotes by wording, author or verdict (verified, plausible, disputed, misattributed).
QuoteCitation cite_quote MLA 9, APA 7 or Chicago 17 citation for a checked quote, with a warning when it's usually credited to someone else.

In an agent:

from langchain.agents import create_agent
from langchain_graciousquotes import QuoteAttributionCheck, QuoteCitation, QuoteSearch

agent = create_agent(
    "anthropic:claude-sonnet-5-5",  # any tool-calling model
    tools=[QuoteAttributionCheck(), QuoteSearch(), QuoteCitation()],
)
agent.invoke({"messages": [{"role": "user",
    "content": "Did Einstein say the definition of insanity is doing the same thing over and over?"}]})
Enter fullscreen mode Exit fullscreen mode

This needs LangChain 1.0+ and Python 3.10+. The tools work on their own from Python 3.9. Every tool also supports ainvoke.

Check every quote in a draft, with no model at all

You don't need an agent to use these. The more useful thing for me turned out to be a plain script: examples/check_draft.py finds the quotes in a Markdown file or a live URL, reads the name the article gives each one ("Emerson told us", "— Seneca"), and checks both.

Here's the run on the repo's sample draft, 22 October 2026 (trimmed):

4 to fix · 1 unverified · 2 OK

FIX        "Do not follow where the path may lead. Go instead where there is no path and leave a trail."
           Your draft credits: Ralph Waldo Emerson
           Record: Re-credited. No: Ralph Waldo Emerson did not say it. It comes from Muriel Strode.
           Fix: Credit Muriel Strode instead of Ralph Waldo Emerson.

FIX        "The only thing we have to fear is fear itself."
           Your draft credits: Mark Twain
           Record: Verified. Yes: President Franklin D. Roosevelt said it.
           Fix: Credit President Franklin D. Roosevelt instead of Mark Twain. Source: First Inaugural Address (1933).

FIX        "The best way to predict the future is to create it."
           Your draft credits: Peter Drucker
           Fix: No reliable source shows Peter Drucker said it. Write "often attributed to Peter Drucker", or cut it.

UNVERIFIED "The quiet hours before dawn belong to people who keep their own promises."
           Fix: No checked record. Keep the quote only if you can cite where it first appeared.

OK         "insanity is doing the same thing over and over again and expecting different results."
           Your draft credits: Albert Einstein (hedged)
           Note: Fine as written: you already present it as an attribution, which matches the record.
Enter fullscreen mode Exit fullscreen mode

The Emerson line is a good example of why this is worth automating: it is widely credited to Emerson, but it comes from Muriel Strode's poem "Wind-Wafted Wild Flowers", printed in The Open Court in August 1903.

The script exits with code 1 when something needs fixing (--strict also fails on CHECK and UNVERIFIED), so it can block a publish step:

python examples/check_draft.py content/posts/new-post.md || exit 1
Enter fullscreen mode Exit fullscreen mode

Note the last OK. The draft says "often attributed to Einstein", which matches the record, so it passes. A checker that flags every hedged attribution gets switched off within a week.

Three design decisions

1. "No record" is not a verdict

The easy failure is a tool that returns nothing, after which the model fills the gap from memory and sounds just as confident. When there's no checked record, the tool says so explicitly and tells the model not to present a remembered author as confirmed. You can see it in the UNVERIFIED line above. It's the same reason a missing value in a spreadsheet shouldn't be read as zero.

2. The tool description is the model's prompt, so be honest about the limits

The author filter in search_checked_quotes matches the person who really said the quote. That means "show me quotes wrongly credited to Mark Twain" can't be answered by searching for author="Mark Twain"; those records are filed under the real author. I found this while testing, and the tool description now says it directly:

The author filter matches the person who really said it, so it cannot list quotes wrongly credited to someone; use check_quote_attribution for a specific quote instead.

If you only document a limit in the README, the model never reads it. Put it in the description.

3. Every result carries its own credit line

Each result ends with Source: Gracious Quotes … Link the record above when you repeat this. That's a terms of use condition, and it also works in the user's favour: a quote verdict without a link to its evidence is just another claim. With the record link, the reader can check the source themselves.

Testing against LangChain's own conformance suite

The package passes langchain-tests (ToolsUnitTests and ToolsIntegrationTests, sync and async) along with its own behaviour tests. Two things worth knowing if you build a tool package:

  • Set asyncio_mode = "auto" in your pytest config. Without it, pytest skips the async cases and the run still looks like a pass.
  • CI runs with no API key on purpose. The API is keyless, so a green build shows that the advertised zero-setup path works for a stranger, not just on my machine. A scheduled run also catches the live API drifting.

A separate CI job builds the create_agent example on Python 3.13 so the guide's code can't quietly break.

Getting listed: file through the web form

LangChain doesn't accept integration PRs. You file an "Integration listing" issue on langchain-ai/docs, a maintainer applies a label, and a bot opens the docs PR. One trap from my first listing (a sister project): an issue created with gh issue create never triggers the automation. It doesn't get the template's labels, so the bot never sees it. Filed again through the web form, it went through in days. For this package: issue filed 9 October, approved and merged on 20 October.

Limits, stated plainly

  • Coverage is what we've checked. Anything outside the records comes back UNVERIFIED rather than guessed.
  • Name detection in check_draft.py is a heuristic. When it isn't sure who a draft credits, it reports CHECK, not a firm verdict.
  • Verdicts can change. When new evidence turns up we update the record and log it on the public corrections page. The full method is on How we source quotes.
  • Rate limit: 40 requests per 10 seconds per IP. A tool that hits it says so in its result instead of raising.
  • Bulk copying or training on the records needs a licence. Fetching records because a user asked is fine, with the credit.

Links

If you paste in a quote and think our verdict is wrong, open an issue with your source. Corrections with a primary source are the most useful thing anyone can send us.

I'm Jeremiah Say, founder and editor of GraciousQuotes. I research quote attributions against original books, speeches and letters, and I maintain this package. I wrote earlier about how the site matches misremembered quotes.

Top comments (0)