DEV Community

Cover image for Checking Borat quotes with Python's difflib: Wikiquote vs the film transcript
Jeremiah Say
Jeremiah Say

Posted on

Checking Borat quotes with Python's difflib: Wikiquote vs the film transcript

Most movie quote pages copy each other. One site types a line from memory, another copies it, and soon "Very nice!" is followed by words nobody in the film ever said.

For our Borat quotes page we used a simple rule: a film line goes on the page only if two separate texts agree on it. The first is Wikiquote. The second is a transcript of the film. A short Python script decides whether they agree.

Of the 97 Borat lines the script checked, 74 passed. This post covers how it decides, why the other 23 failed, and a mistake the check missed that I found while writing this.

The two witnesses

Witness 1: Wikiquote. We fetch each film's page as wikitext (action=raw) and parse it into JSON: one record per line, with the speaker and the film. The wording on our page is copied from this file, never retyped.

Witness 2: a transcript. For Borat (2006) we used the Springfield! Springfield! transcript. For Borat Subsequent Moviefilm (2020) we used the Scraps from the Loft transcript. Both were read from Wayback Machine copies (2006, 2020), so anyone can open the same text we checked.

Neither source is perfect. Transcripts are fan-made, and Wikiquote is edited by volunteers. But they come from different people. When both say the same thing, a typo is far less likely to slip through.

Step 1: make both texts comparable

Raw string comparison fails at once. Wikiquote writes Borat's accent as "name-a" and "is-a"; transcripts write "she's borin'". So both texts go through the same tokenizer:

import re

CONTRACTIONS = {"n't": " not", "'re": " are", "'ll": " will", "'ve": " have",
                "'m": " am", "'d": " would", "'s": " s"}

def toks(s):
    s = s.lower().replace("’", "'").replace("‘", "'")
    for a, b in CONTRACTIONS.items():
        s = s.replace(a, b)
    words = [re.sub(r"[^a-z0-9]", "", w)
             for w in re.split(r"[\s\-—–…]+|\.\.\.", s)]
    # "boring" -> "borin" so it meets the transcript's "borin'"
    return [w[:-1] if len(w) > 4 and w.endswith("ing") else w
            for w in words if w]
Enter fullscreen mode Exit fullscreen mode

Three small choices do most of the work:

  • Expand contractions. "I'm" and "I am" become the same tokens.
  • Split on dashes. "name-a" becomes name, a. Borat's accent is then one extra short word, not a different word.
  • Drop the final g of "-ing" words. Then "boring" and "borin'" both become borin. The length check leaves "king" and "ring" alone.

toks("She is-a boring. I'm-a new in town") gives ['she', 'is', 'a', 'borin', 'i', 'am', 'a', 'new', 'in', 'town'].

Step 2: find the best place in the transcript

A quote is a few words. The transcript is about 8,200 words (2006) or 12,700 words (2020), with no timestamps that line up with Wikiquote. So the script slides a window along the transcript and keeps the best match:

from difflib import SequenceMatcher

def check(quote, transcript_tokens):
    q, c = toks(quote), transcript_tokens
    n, best = len(q), 0
    edges = set(q[:3]) | set(q[-3:])
    for i in range(max(1, len(c) - n // 2)):
        # cheap filter: start near a word from the quote's first or last
        # three words, and still try every 4th position as a fallback
        if c[i] not in edges and i % 4:
            continue
        window = c[max(0, i - 2): i + n + 6]
        m = sum(b.size for b in SequenceMatcher(
            None, q, window, autojunk=False).get_matching_blocks())
        best = max(best, m)
        if m == n:
            break
    return n, best
Enter fullscreen mode Exit fullscreen mode

get_matching_blocks() counts the quote's words found in the same order in the window. A few extra transcript words ("uh", a name, a repeat) cost nothing. A missing or different word costs one.

autojunk=False is a safety setting. By default, SequenceMatcher treats very common tokens as junk once the second sequence has 200 or more items. These windows are much shorter, so it would rarely fire, but turning it off means it never can.

Step 3: a pass bar

def passes(n, matched):
    return matched / n >= 0.87 and (n - matched) <= max(2, n // 10)
Enter fullscreen mode Exit fullscreen mode

Two conditions, because either alone fails:

  • A ratio alone is too strict for short lines. One wrong word in a five-word line is 80%.
  • A miss count alone is too loose for long lines. Two misses in a 12-word line might be the two words that carry the joke.

The page build stops if a line fails. Every film quote goes through an assert before the page JSON is written (one line has a noted exception: the transcript spells Borat's drawn-out "niiiice" as "ni-i-ice"). Two more checks run with it: the excerpt we show must sit inside its Wikiquote line, and the speaker must match.

What failed, and why

Lines from 4 to 130 words were checked: 97 across both films. 74 passed and 23 didn't. Reading the 23 by hand, they fall into four groups:

Why it failed Example Words matched
Short line, one word differs (11 of the 23 are 7 tokens or fewer) "No! No, you cannot do that." 4 of 6
The two witnesses spell the accent or a name differently Wikiquote "This-a my wife Oksana. She is-a boring." / transcript "this my wife Oxana she's borin'" 5 of 9
Wikiquote includes a stage direction "I'm-a new in town. (A chicken falls out of his briefcase)" 9 of 18
The line isn't in the transcript at all "Only men and bears are allowed inside car." 2 of 8

A failure doesn't mean Wikiquote is wrong. It means two texts don't yet agree, so the line stays off the page. Some of those lines are fine, and a third text would let us use them.

Excerpts: quoting the part both witnesses agree on

The sequel opens with Borat in a labor camp:

Jak sie masz? My name-a Borat. My life is nice, NOT! But how I end up like this?

The full line scores 16 of 19. The transcript spells the greeting as one word ("jangshemash"), so three tokens miss. Without the greeting, the excerpt scores 16 of 16. That's the version on the page. The build asserts that the excerpt is inside the Wikiquote line, so we can trim but never invent.

One witness means it's left out

Some famous lines appear in only one text. "King in the castle" and "Wawaweewa" are in a transcript but not on Wikiquote, and the 2006 ending line ("if you chase a dream...") was in neither of our texts. They aren't on the page. That's the cost of the rule.

The bug the check didn't catch

"Great success!" isn't on Wikiquote as its own line. It's Borat's catchphrase, scattered through both films. So we checked it straight against both transcripts (a 2 of 2 match in each) and wrote a short note on how often he says it.

The note said five times in the 2006 film. While writing this post I counted again, across line breaks this time:

import re
text = " ".join(open("borat_2006.txt").read().split())
print(len(re.findall(r"great\W+success", text, re.I)))  # 4
Enter fullscreen mode Exit fullscreen mode

Four. The note had been typed by hand, not computed from the text. The alignment checked the words, but nothing checked the number. We fixed the page and the evidence record on 20 November 2026, and it now reads "four times in the 2006 film's transcript".

The lesson I'm taking from this: a number in a sentence is a claim too. If it was counted, the count belongs in code, next to the check that backs it.

Across the site

The same script runs on every film and character page we've built this way. On the 11 pages whose results are saved, it checked 5,044 Wikiquote lines and 3,599 passed (71%). The pass rate depends on the transcript as much as on Wikiquote: a fan transcript with gaps fails good lines. That's why a failed line is held back, not marked false.

Try it on your own data

The method needs nothing beyond the standard library:

  1. Tokenize both texts with the same function, tuned to the speaker's spelling quirks.
  2. Slide a window and count in-order matches with SequenceMatcher(..., autojunk=False).
  3. Pass on a ratio and a miss cap.
  4. Make the build fail on a miss, so a person has to look.
  5. Compute every count you publish. Don't type it.

You can see where each line came from on the Borat quotes page: every quote has a "See the full check" link to both witnesses. Our sourcing method explains the labels, and there are more checked lines in famous movie quotes.


I'm Jeremiah Say, founder and editor of GraciousQuotes. I check quote attributions against original books, speeches, letters and transcripts. Earlier: matching misremembered quotes and a LangChain tool that checks attributions.

Top comments (0)