DEV Community: Philipp Primisser

My keyword matcher couldn't find "ChatGPT" in Japanese posts. The bug was `\w`.

Philipp Primisser — Fri, 16 Oct 2026 04:06:44 +0000

I've been building a small tool that watches Bluesky and Mastodon for posts that mention a keyword: your product, your project, your name. Matching a keyword in a post sounds like the trivial part. It wasn't.

The first surprise: who is actually posting

Before writing any matching code I wanted to know which languages matter. So I pulled the latest ~100 Bluesky search results per language for two keywords and measured how fast they came in.

Keyword	en	ja	es	pt	fr	de	ko
chatgpt (posts/hour)	47.5	11.9	2.2	2.2	2.2	2.1	0.3
netflix (posts/hour)	51.1	3.9	3.5	2.6	3.2	1.5	0.1

English dominates, no surprise. But for "chatgpt", Japanese was a clear second, more than five times Spanish or Portuguese. That's a tiny sample on one day, so don't quote it as a market study. It was enough to make me test Japanese posts properly, and that's where the matcher broke.

Trap 1: `\b` and `\w` think Japanese is one long word

My matcher used word boundaries so that "apify" doesn't match "apifying", something like (?<!\w)keyword(?!\w). In Python 3, \w is Unicode-aware, which sounds good, until you remember that hiragana, katakana and kanji are all word characters.

Japanese doesn't put spaces between words. In ChatGPTで使う the character after "ChatGPT" is で, which is \w, so there is no boundary and no match. Every Japanese mention of an English product name was silently dropped. Unit tests with English text passed. Only a real run with languages: ["ja"] showed the misses: the search API returned the posts, my filter threw them away.

The fix: a boundary is any non-word character or a character from a script that is written without spaces (CJK, kana, Hangul, Thai, Lao, Myanmar, Khmer).

_NO_SPACE_SCRIPTS = re.compile(r'[\u3040-\u30ff\u3400-\u4dbf\u4e00-\u9fff\uf900-\ufaff'
                               r'\uac00-\ud7af\u0e00-\u0e7f\u0e80-\u0eff\u1000-\u109f'
                               r'\u1780-\u17ff\uff66-\uff9f]')

def _boundary(ch: str) -> bool:
    return not ch or not re.match(r'\w', ch) or bool(_NO_SPACE_SCRIPTS.match(ch))

Keywords that are themselves written in those scripts are matched as plain substrings, because there's no word boundary to look for.

Trap 2: "ignore accents" can turn か into が

I wanted an option so that eleicao matches eleição and Muller matches Müller. The textbook way is NFD normalization, then strip all combining marks.

The problem: Japanese dakuten (the two little strokes that turn か into が) are combining marks too (U+3099/U+309A). Strip all combining marks and か and が become the same character, which changes the word. So I only strip U+0300–U+036F, the combining diacritics used by Latin, Greek and Cyrillic:

_DIACRITICS = re.compile(r'[\u0300-\u036f]')

def fold(text, ignore_accents=False):
    t = unicodedata.normalize('NFKC', text or '').casefold()
    if ignore_accents:
        t = unicodedata.normalize('NFC', _DIACRITICS.sub('', unicodedata.normalize('NFD', t)))
    return t

Trap 3: `lower()` is not enough

NFKC + casefold() instead of lower() handles full-width letters (ＣｈａｔＧＰＴ, common in Japanese posts), ligatures, and German ß → ss. Small thing, but full-width brand names were another source of silent misses.

Mastodon is a different problem

On Mastodon, full-text search generally needs an account, so the tool reads public hashtag timelines on a list of servers. In my test 9 of 10 big servers answered normally; one (chaos.social) returned HTTP 422 for the tag timeline, so the tool skips servers like that and lists them in the run summary instead of failing.

What I ended up with

The monitor runs on a schedule and only returns posts it hasn't delivered before, plus an Excel/CSV file and a plain-text digest you can send by email or Slack. It only reads public posts and needs no login. If you want to try it: Bluesky & Mastodon Keyword Monitor on Apify. It's pay-per-use ($0.003 per run + $0.001 per matching post).

If you match keywords in user-generated text, test with Japanese (and Thai) early. English-only test data hides exactly this kind of bug.

I measured my OCR pipeline on 36 old book scans. The phone-photo fix was one division.

Philipp Primisser — Fri, 16 Oct 2026 04:06:08 +0000

I've been building a PDF-to-text tool that also has to cope with scanned pages. At some point "it works on my test PDFs" stopped meaning anything, so I set up a proper test: 36 public-domain book pages from Wikimedia Commons (1700s to 1900s) in German, English, Spanish, French, Hindi and Portuguese, six per language. Wikisource has proofread transcriptions of these pages, so I could score the OCR output with character error rate (CER) instead of squinting at it.

The engine is plain Tesseract 5 at 300 dpi. Everything below is about what happens around it.

One page is out of the averages: a handwritten Portuguese manuscript. Tesseract produced nothing usable (CER 100 %), and no preprocessing trick changes that. The numbers below are over the other 35 printed pages.

The baseline was fine. The phone photos weren't.

On the original scans the final pipeline lands at a mean CER of 5.5 % (median per language between 0.9 % for Portuguese and 6.8 % for Hindi). Old typefaces, long ſ, stains: a few wrong characters per hundred.

Then I simulated what people actually upload: a phone photo of a page. Uneven lighting, lower contrast, blur, heavy JPEG compression and a 2° skew. Same 35 pages, same pipeline: mean CER 20.8 %. With plain global (Otsu) thresholding and only deskew it was even worse, about 50 %. Whole regions of the page vanished because the shadowed half was darker than the "ink" threshold.

The obvious fix is adaptive thresholding (Sauvola, which Tesseract has built in via thresholding_method=2). That brought the phone photos from ~50 % to ~15 %. Better, but still bad, and it had a side effect I'll get to.

The fix that worked: divide by the background

What helped was boring and old: estimate the paper brightness and divide by it. Shrink the page, take a max filter so the text disappears and only the paper remains, blur it, scale it back up, divide.

import numpy as np
from PIL import Image, ImageFilter, ImageOps

def unevenness(im: Image.Image) -> float:
    """Spread of paper brightness over an 8x8 grid. 0 = evenly lit."""
    small = np.asarray(im.resize((256, 256), Image.BILINEAR), dtype=np.float32)
    cells = small.reshape(8, 32, 8, 32).transpose(0, 2, 1, 3).reshape(64, -1)
    paper = np.percentile(cells, 90, axis=1)          # brightest 10% of each cell ~ paper
    return float(np.percentile(paper, 95) - np.percentile(paper, 5))

def flatten(im: Image.Image) -> Image.Image:
    """Divide by a blurred background estimate so uneven light becomes white paper again."""
    small = im.resize((max(1, im.width // 8), max(1, im.height // 8)), Image.BILINEAR)
    bg = (small.filter(ImageFilter.MaxFilter(9))      # remove the text, keep the paper
               .filter(ImageFilter.GaussianBlur(6))
               .resize(im.size, Image.BILINEAR))
    a = np.asarray(im, dtype=np.float32)
    b = np.maximum(np.asarray(bg, dtype=np.float32), 1.0)
    out = np.clip(a / b * 255.0, 0, 255).astype(np.uint8)
    return ImageOps.autocontrast(Image.fromarray(out), cutoff=0.5)

im = Image.open("page.png").convert("L")
if unevenness(im) > 30:   # only touch pages that are actually unevenly lit
    im = flatten(im)

The > 30 gate matters. Evenly lit scans don't get touched at all, so they can't get worse.

Result on the phone simulations: 20.8 % → 5.7 %, basically back to the 5.5 % of the originals. The same pages turned by 90° end at 5.1 %.

Why Sauvola is only a retry

Once Sauvola had rescued the phone photos, I was tempted to just use it everywhere. On the 35 original scans that didn't change the mean (5.7 % either way), but the median went from 3.8 % to 4.4 % and a clean Portuguese page roughly doubled its errors (1.0 % → 1.9 %). Clean pages don't need local thresholding, and it adds noise around faint letters.

So now it's a retry: Otsu first, and only if Tesseract's mean confidence is below 90 do I also run Sauvola and keep whichever result has more confidently recognized characters.

Throwing away lines Tesseract doesn't believe

Old books have stains, show-through from the other side, decorations. Tesseract happily "reads" those as lines of garbage. Its TSV output gives a confidence per word, so I average per line and drop lines below 40:

NOISE_CONF = 40

def is_noise(words: list[tuple[str, float]]) -> bool:
    """A line Tesseract itself barely trusts: stains, show-through, page edges, pictures."""
    cs = [c for _, c in words if c >= 0]          # conf = -1 means "no word here"
    return bool(cs) and sum(cs) / len(cs) < NOISE_CONF

# words come from: tesseract page.png stdout tsv  (level 5 rows = words)
lines = [join(words) for words in lines_of_paragraph if not is_noise(words)]

That was the single biggest win on the originals: mean CER 6.8 % → 5.5 %. One French page went from 9.4 % to 3.3 %, another from 47.9 % to 8.8 % (it had a large illustration that came out as a wall of junk characters).

Small things that cost me time

Fraktur needs its own model. For the German blackletter pages, the regular deu model gave 11–25 % CER, the frk model 4–14 %. Tesseract's script detection recognizes Fraktur reliably, so switching automatically is cheap.
Rotate: 90 from Tesseract's OSD means "turn it clockwise by 90". PIL's rotate() is counter-clockwise, so it's im.rotate(-90, expand=True). I had it backwards at first and turned sideways pages upside down.
Don't trust OSD's script guess on bad pages. On a poor scan it called Latin text "Cyrillic" at confidence ~1. I now only trust the script if its confidence is ≥ 3; otherwise I do a quick English pass and guess the language from the text.
Born-digital PDFs have their own trap. With pdfplumber's default tolerance, LaTeX papers came out as "Thedominantsequence…". extract_text_lines(..., x_tolerance_ratio=0.15) fixed the word spacing.
License check before picking a PDF library. PyMuPDF is great but AGPL, which I didn't want in a paid tool, so it's pdfplumber + pypdfium2 + Tesseract, all under permissive licenses.

What it costs to run

It runs on Apify with 1 GB of memory, which is a quarter of a CPU, and Tesseract on one thread. A dense OCR page takes roughly 18–25 seconds there, about $0.0011–0.0014 per page in compute (measured over three 10-page runs). A page that already has a text layer costs around $0.00004, which is why the tool only OCRs pages that need it.

Caveats

36 pages is a small sample, and the phone photos are simulated, not real photos. Old book typography is also harder than a modern invoice: on a clean German EU regulation rendered at 300 dpi I get 0.3–1.7 % CER. If you have a set of real phone photos with ground truth, I'd genuinely like to know how this holds up.

What do you use against uneven lighting: background division, morphology, or something learned?

The pipeline is available as a pay-per-use tool on Apify if you want to try it on your own PDFs: PDF to Text, Markdown & OCR.

Free EU tender alerts from the TED API (one Python file, no account)

Philipp Primisser — Fri, 09 Oct 2026 01:54:23 +0000

The EU publishes thousands of public procurement notices every week on TED (Tenders Electronic Daily). Commercial tender alert services usually charge a subscription. If you already know which CPV codes you bid on, the official Search API is enough for a personal alert.

I put that into one Python file with no dependencies beyond the standard library: ted-tender-alerts (MIT).

python ted_alerts.py --cpv 72 48 --country AUT --days 7 --lang de

22 matching notices, 22 new.
- Commvault
  … | AUT | 21,000,000 EUR | deadline 2026-11-09 (32 d)
  https://ted.europa.eu/de/notice/-/detail/696492-2026
- MA 9 – Digitalisierung 2027-2031
  Magistrat der Stadt Wien … | AUT | deadline 2026-11-17 (40 d)
  …

(Real output from 8 October 2026. Run it again and you get 22 matching notices, 0 new.)

What the API gives you

TED Search API v3 is anonymous. One POST to https://api.ted.europa.eu/v3/notices/search with a query string in TED's expert-search language:

query = (
    "publication-date>=20261001"
    " AND buyer-country IN (AUT DEU)"
    " AND notice-type IN (cn-standard cn-social cn-desg subco qu-sy)"
    " AND (classification-cpv=72* OR classification-cpv=48*)"
    " SORT BY publication-date DESC"
)
body = {"query": query, "fields": [...], "page": 1, "limit": 100,
        "paginationMode": "PAGE_NUMBER", "scope": "ALL"}

CPV prefixes work: 72 matches every code under 72000000 (IT services). Countries use ISO alpha-3 (AUT, DEU). --print-query prints the string so you can paste it into TED's expert search and compare.

The script is gentle with the API: it pauses 0.5 s between result pages and stops after 20 pages (100 notices each) by default.

Only the new ones

A state file (ted_state.json) remembers publication numbers already reported. Every run compares the current result to that set and prints only the difference. Notices older than 120 days drop out of the state, so the file stays small.

Same idea for an Atom feed of the latest 100, a CSV you append to, a Markdown table, or an e-mail through your own SMTP server (SMTP_HOST, MAIL_TO, … — the password is never printed or written).

Run it daily for free on GitHub Actions

The repository includes an example workflow (examples/daily-ted-alerts.yml). Copy it to .github/workflows/ in your own copy of the repo and it runs every weekday morning, commits the state and the Atom feed back, and optionally e-mails you. Edit the CPV/country filters in the YAML; add SMTP secrets if you want mail. Point a feed reader at the raw URL of data/feed.xml (public repos) or just open data/latest.md.

Finding your CPV codes

Look at five tenders you would have bid on and note their CPV codes (they're on every TED notice). Shorter prefixes catch more: 72 is all IT services, 7222 only IT consultancy. Start broad, then narrow once you see the noise.

Legal note

Data comes from TED – Tenders Electronic Daily, © European Union. Notices may be reused free of charge with attribution (TED legal notice). The script adds the attribution line to every output.

Try it

git clone https://github.com/philippprimisser-max/ted-tender-alerts
cd ted-tender-alerts
python ted_alerts.py --cpv 72 --country AUT --days 3

AI-assisted: a model helped with the wording. The sample run and the 22-notice count are from my own machine on 8 October 2026. Not affiliated with the EU Publications Office.

Word-by-word captions with Python and ffmpeg, no CapCut watermark

Philipp Primisser — Fri, 09 Oct 2026 01:39:29 +0000

Short-form video lives or dies on captions. CapCut and a dozen web tools will do the "word lights up as you say it" style for you, usually with a watermark or a free-minute limit. I wanted the same look without an account, so I wrote a small offline script: whisper-word-captions (MIT). This post is about how the trick works, so you can change it.

python wordcaptions.py clip.mp4
# -> clip_captioned.mp4  and  clip.ass

Three pieces

faster-whisper with word_timestamps=True → every word gets its own start and end.
Group words into short captions (max 3 by default; also break after .?! or a pause over 0.6 s).
Write an Advanced SubStation Alpha (.ass) file with one Dialogue event per spoken word, then burn it into the picture with ffmpeg's subtitles filter (libass).

The interesting part is step 3.

The ASS trick

ASS lets you override style mid-line with curly-brace tags. \c changes the text colour (ASS colours are &HAABBGGRR, blue-green-red), \fscx/\fscy scale it, \r resets to the style. For a three-word caption you write three events, one per word:

Dialogue: 0,0:00:03.20,0:00:03.52,Word,,0,0,0,,{\c&H0000D4FF\fscx115\fscy115}THAT{\r} ALL MEN
Dialogue: 0,0:00:03.52,0:00:03.68,Word,,0,0,0,,THAT {\c&H0000D4FF\fscx115\fscy115}ALL{\r} MEN
Dialogue: 0,0:00:03.68,0:00:03.90,Word,,0,0,0,,THAT ALL {\c&H0000D4FF\fscx115\fscy115}MEN{\r}

Each event lasts until the next word starts. That way nothing flickers during short pauses, and the viewer always sees the whole caption with only the current word highlighted. In Python that's just a nested loop:

for group in groups:
    texts = [escape_ass(w.text.upper()) for w in group]
    for i, w in enumerate(group):
        end = group[i + 1].start if i + 1 < len(group) else max(w.end, w.start + 0.05)
        parts = []
        for j, t in enumerate(texts):
            if j == i:
                parts.append(f"{{\\c{hl}\\fscx115\\fscy115}}{t}{{\\r}}")
            else:
                parts.append(t)
        events.append(f"Dialogue: 0,{ass_time(w.start)},{ass_time(end)},Word,,0,0,0,,{' '.join(parts)}")

PlayResX/PlayResY in the .ass header match the video size, so the same file works for 1080×1920 Shorts and for landscape. Font size is a percentage of the shorter side (6.5 % by default), with a bit more margin at the bottom so the caption sits above the like/comment buttons.

Burning it in

ffmpeg -i clip.mp4 -vf "subtitles=clip.ass" -c:v libx264 -crf 20 -c:a copy clip_captioned.mp4

Audio is copied, video is re-encoded once. If you use a non-system font, pass a fonts directory:

-vf "subtitles=clip.ass:fontsdir=./fonts"

Paths with backslashes or drive letters need escaping for the filter (C:\… → C\:/…). The script does that for you.

Edit without re-transcribing

Speech recognition gets names and jargon wrong. The .ass is plain text, so open it, fix the words, and re-burn:

python wordcaptions.py clip.mp4 --render-only

That takes seconds because Whisper doesn't run again. --srt also writes a normal subtitle file if you want something for YouTube's upload form.

Install

Python 3.10–3.13 and ffmpeg (with libass, which the common builds include).

# Windows: winget install Gyan.FFmpeg
# macOS:   brew install ffmpeg
# Debian:  sudo apt install ffmpeg
git clone https://github.com/philippprimisser-max/whisper-word-captions
cd whisper-word-captions
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

The first run downloads the Whisper model (small ≈ 490 MB). requirements.txt pins av<19, because PyAV 19 broke faster-whisper 1.2.1 on a fresh install in October 2026.

Defaults that matter: --model small (better for non-English than base) and highlight colour #FFD400. --threads is capped at 4. On the bench from the gap checker (base, int8, 5 clips × 3 runs, bench/run_table.sh, 9 October 2026) 8 threads had gaps in 13 of 15 runs (longest 60.5 s), 4 threads in 1 of 15 (longest 28 s) and 1 thread in 1 of 15 (longest 6 s). float32 at 8 threads was 0 of 15. Fewer threads reduce the gaps. They don't remove them. Check the .ass, or use faster-whisper-gap-check.

Measured on an 8-core Linux server: a 15-second 1080×1920 clip took 6.1–6.7 s end to end with small (model load, transcription and burn; two runs, model already downloaded). The clip is made from the repo's test audio, so you can repeat it; the commands are in the README. Your laptop will differ.

AI-assisted: a model helped with the wording. The script was tested and the timing is from my own server (9 October 2026). Demo audio: LibriVox recording of the Gettysburg Address (public domain). Not affiliated with CapCut, TikTok, YouTube, Instagram, OpenAI or the ffmpeg project.

Podcast to text in Python: transcribe any episode from its RSS feed

Philipp Primisser — Fri, 09 Oct 2026 01:16:30 +0000

I wanted transcripts of a few podcast episodes for notes and search. The options I found were either a web service with a monthly plan, or a Whisper tutorial that starts with "first, download the MP3". I didn't want to do the download part by hand every time, so I wrote a small script that takes whatever link I have, an RSS feed, an Apple Podcasts link or a direct audio URL, and gives me .txt, .srt, .vtt and .json.

It's on GitHub: podcast-to-text (MIT). This post walks through the parts that turned out to be interesting.

python podcast_to_text.py "https://podcasts.apple.com/us/podcast/hacker-public-radio/id281699640" --search "wl-copy"

Apple Podcasts link -> RSS feed: https://hackerpublicradio.org/hpr_rss.php
Episode: HPR4743: wl-copy (2026-10-07)
Transcribing with faster-whisper 'base' (4 threads) ...
Done: 15.7 min audio in 18 s (0.02 s per audio second), language en.

Step 0: maybe you don't need to transcribe at all

Podcasting 2.0 added a <podcast:transcript> tag to RSS. Some hosting platforms generate transcripts and put the link right into the feed. So before burning CPU, check the feed.

I was curious how common that is, so I looked at the Apple top-chart podcasts for the US, UK, Germany and Austria (109 feeds I could read, checked on 8 October 2026; the script and the raw results are in the repo's research/ folder):

Chart	Newest episode has a transcript tag	At least one episode has one
US	5 / 49	5
UK	2 / 25	3
Germany	8 / 25	15
Austria	3 / 10	5
Total	18 / 109	28

So for roughly one in six popular shows the transcript is already there. German shows were ahead in this sample, mostly because many of them are hosted on Podigee: 18 of the 28 hits were Podigee feeds (a few shows chart in both Germany and Austria, so they count twice). The others were on Omny, Buzzsprout, Flightcast, Captivate and two smaller hosts. Most of them were WebVTT, some JSON, a few SRT or plain text. The script prefers JSON, then SRT, then VTT, then plain text, downloads it and stops. --force transcribes anyway.

Reading the tag is a few lines with the standard library:

PODCAST_NS = "https://podcastindex.org/namespace/1.0"

for item in channel.findall("item"):
    transcripts = [
        {"url": t.get("url"), "type": (t.get("type") or "").lower()}
        for t in item.findall(f"{{{PODCAST_NS}}}transcript")
        if t.get("url")
    ]

Step 1: Apple Podcasts links are just a pointer to an RSS feed

Most people share Apple Podcasts links, not RSS URLs. Apple's public lookup API returns the show's own feed URL for the numeric ID in the link:

show_id = re.search(r"/id(\d{5,})", urlparse(url).path).group(1)
data = json.loads(http_get(f"https://itunes.apple.com/lookup?id={show_id}&entity=podcast"))
feed = data["results"][0]["feedUrl"]

If the link points to a single episode (?i=1000…), a second lookup with entity=podcastEpisode gives you that episode's GUID, which you match against the feed. The audio then comes from the publisher's server, exactly as in any podcast app. Apple-exclusive subscriber shows have no public feed, and the script says so instead of trying anything clever.

Step 2: pick the episode

--list prints the latest 30 episodes, with a marker for those that already have a transcript. -e 2 takes the second newest, --search "some words" takes the newest episode whose title contains all of them. Parsing is plain xml.etree; I didn't add feedparser because the script only needs title, date, GUID, enclosure and the transcript tag.

One thing I kept: an honest user agent (podcast-to-text/0.1 (+github URL)). A few hosts block unknown clients. Pretending to be a browser would get around that, but if a publisher doesn't want automated downloads, that's their call.

Step 3: transcribe with faster-whisper

model = WhisperModel(model_name, device="cpu", compute_type="int8",
                     cpu_threads=min(4, os.cpu_count() or 1))
segments, info = model.transcribe(str(audio), language=language, beam_size=1,
                                  vad_filter=True, condition_on_previous_text=False)

A few choices here, and why:

vad_filter=True skips silence and music beds, which saves time and avoids some of Whisper's "Thank you for watching" hallucinations on silent parts.
condition_on_previous_text=False makes it less likely that one bad segment drags the following ones into a repetition loop.
cpu_threads capped at 4. On one 8-core machine, int8 with the base model dropped speech in 13 of 15 runs at 8 threads (longest 60.5 s), 1 of 15 at 4 threads (longest 28 s) and 1 of 15 with a single thread (longest 6 s). That's 5 clips × 3 runs, from bench/run_table.sh on 9 October 2026. Fewer threads make the gaps rarer. They don't get rid of them. float32 at 8 threads was clean, 0 of 15, which is why the script has --compute-type float32. I wrote the check up separately: faster-whisper-gap-check.

How fast is it? On an 8-core Linux server with 4 threads, not counting model loading: the 15.7-minute HPR episode above took 18 s with base. The one-minute Gettysburg Address from the repo's tests took 1 s with base and 9–10 s with small (python podcast_to_text.py tests/fixtures/gettysburg.mp3 -m small --language en). A laptop will be slower; I'm not going to guess by how much.

On quality: for clear English, base was fine in my tests; for German and other languages I'd start with small. Pass --language if you know it. Auto-detection can guess wrong: in one run without it, small took the English test clip for Russian and wrote the whole transcript in Russian.

Step 4: four output formats

Segments come back as start, end, text. From that:

.txt with a paragraph break after pauses over 2 seconds, which makes long transcripts much easier to read
.srt and .vtt for players and video editors
.json with the segments plus metadata (model, detected language, audio length, processing time)

The SRT timestamp function is the only fiddly bit:

def ts(seconds, sep=","):
    ms = int(round(max(seconds, 0) * 1000))
    h, ms = divmod(ms, 3_600_000)
    m, ms = divmod(ms, 60_000)
    s, ms = divmod(ms, 1000)
    return f"{h:02d}:{m:02d}:{s:02d}{sep}{ms:03d}"

Install and run

git clone https://github.com/philippprimisser-max/podcast-to-text
cd podcast-to-text
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
python podcast_to_text.py https://hackerpublicradio.org/hpr_mp3_rss.php --list

No ffmpeg needed, faster-whisper decodes audio through PyAV. One trap: on a fresh install in October 2026, PyAV 19 broke faster-whisper 1.2.1 with open() got an unexpected keyword argument 'metadata_errors', so requirements.txt pins av<19.

What it doesn't do

No speaker labels, no summaries, no private feeds unless you have a personal feed URL you're allowed to use. Machine transcripts still get names and jargon wrong, so read before you quote.

AI-assisted: a model helped with the wording. The code was tested and all numbers come from runs on my own server (8 and 9 October 2026) and can be repeated with the commands in the repositories. Test audio: LibriVox (public domain) and Hacker Public Radio (CC BY-SA 4.0). Not affiliated with Apple; "Apple Podcasts" is a trademark of Apple Inc.

faster-whisper int8 dropped up to 60 s of speech. float32 didn't

Philipp Primisser — Fri, 09 Oct 2026 00:27:09 +0000

I run a small podcast transcription tool, and while testing German audio I noticed that some transcripts were missing whole passages. Not wrong words. Twenty, thirty, sometimes close to sixty seconds of speech, gone. The timestamps simply jumped ahead, the text still read fine, and nothing in the logs said anything was wrong.

Here's one of them. A one-minute excerpt of a LibriVox reading of Theodor Storm's Der Schimmelreiter, transcribed with faster-whisper base, int8, 8 threads:

1
00:00:30,690 --> 00:00:32,110
Enkel singelit.

The first subtitle starts at 0:30. The narrator starts reading at 0:01.

What I measured

Setup: faster-whisper 1.2.1, CTranslate2 4.8.2, model base, beam size 1, VAD on, condition_on_previous_text=False, on one 8-core x86-64 Linux server (os.cpu_count() = 8). Five public clips of about 60 seconds (three English, two German: LibriVox recordings and one Hacker Public Radio episode), three runs per configuration and clip, because the problem doesn't happen every time. A run counts as "with a gap" if at least 5 seconds of detected speech have no transcript segment.

Config	Runs with a gap ≥ 5 s	Longest gap	Avg time per clip
int8, 1 thread	1/15	6.0 s	4.7 s
int8, 4 threads	1/15	28.0 s	4.0 s
int8, 8 threads	13/15	60.5 s	2.4 s
float32, 8 threads	0/15	–	3.8 s

(9 October 2026. The machine wasn't idle during the run, so treat the times as rough.)

int8 with 8 threads lost audio in almost every run, and when it did, it lost a lot: the 60.5-second gap was in a 61.5-second clip. Fewer threads made it much rarer, but not zero: one run with 4 threads missed 28 seconds, and one run with a single thread missed 6 seconds. The only configuration without a single gap was float32.

The dropping is random. A day earlier I ran the same clips with the same tool and got gaps in 4 of 15 int8/8-thread runs and none anywhere else. I didn't record the exact flags of that run, so the table above is the one to go by; the raw output of both is in the repository.

I don't know the root cause. My guess would be something in CTranslate2's int8 path under higher thread counts on this CPU, but that's a guess and I haven't dug into it. This is one machine. I haven't tested other CPUs, other model sizes or other CTranslate2 versions.

What I'd do now

If you can't check your transcripts, use float32:

from faster_whisper import WhisperModel

model = WhisperModel("base", device="cpu", compute_type="float32")

In my run it was the only clean configuration, and at 8 threads it took 3.8 s per one-minute clip, against 2.4 s for int8 with 8 threads and 4.0 s for int8 with 4 threads. So it costs some speed compared to int8 with many threads, but not compared to int8 with a capped thread count.

If you stay on int8, capping the threads (cpu_threads=min(4, os.cpu_count() or 1)) cut the gaps from 13 of 15 runs to 1 of 15 here. That's a big improvement, not a guarantee. Check the output.

Check your own transcripts

The nasty part is that you don't notice. A transcript with a missing half minute looks like a normal transcript. So I turned the checking into a tool: faster-whisper-gap-check (MIT).

It runs Silero VAD (which ships with faster-whisper) over the audio, finds where people are speaking, and lists every stretch of speech that no transcript segment covers:

python gapcheck.py check s3_schimmelreiter.mp3 results/schimmelreiter_int8_8threads.srt

Audio 1:01.0 | speech detected 56 s | transcript segments 20
Possibly dropped: 1 stretch(es), 29 s of speech (52 %):
  0:00.9 - 0:29.7  (29 s)

It reads .srt, .vtt and JSON with start/end segments, so it works on output from any Whisper flavour or API, not only faster-whisper. Exit code 1 if something's missing, so you can drop it into a batch job.

The core is just interval arithmetic:

def uncovered(speech, covered, pad=1.0):
    cov = merge([(max(0.0, a - pad), b + pad) for a, b in covered])
    result = []
    for s0, s1 in speech:
        cur = s0
        for c0, c1 in cov:
            if c1 <= cur or c0 >= s1:
                continue
            if c0 > cur:
                result.append((cur, c0))
            cur = max(cur, c1)
            if cur >= s1:
                break
        if cur < s1:
            result.append((cur, s1))
    return result

And there's a repro mode that transcribes the same file several times per configuration and checks every run. The table above comes from bash bench/run_table.sh, which cuts the five clips from archive.org with ffmpeg and runs, per clip:

python gapcheck.py repro CLIP.mp3 --language en --threads 1 4 8 --runs 3 --json results/my-run/gap_CLIP.json
python gapcheck.py repro CLIP.mp3 --language en --threads 8 --compute-types float32 --runs 3 --json results/my-run/gap_CLIP_f32.json

(--language de for the German clips.) If you run faster-whisper on a many-core CPU, I'd be curious what you get. Use a clip with continuous speech (an audiobook chapter is ideal) and do at least three runs per config; one clean run proves nothing.

Caveat on the checker: music with vocals, laughter or crosstalk counts as speech for the VAD, so a "gap" over a jingle can be fine, and a short gap like the 6-second one above is worth listening to before you call it a bug. Treat the output as "go listen to these timestamps", not as a verdict.

What I took away from it

Test with the thread count and compute type you deploy with. A single thread also had one short gap in my test, so a low thread count alone is no guarantee.
Run quality tests more than once. faster-whisper output varies between runs. My two runs of the same experiment gave 4/15 and 13/15 for the same configuration.
Check coverage, not only accuracy. Word error rate against a reference catches this, but most people don't have a reference. Comparing against VAD speech regions needs nothing but the audio.

The free podcast-to-text script caps threads at 4 by default and has --compute-type float32 if you want to be safe.

AI-assisted: a model helped with the wording. The measurements are from my own server (8 and 9 October 2026), and the raw data and the script are in the repository. Test audio: LibriVox recordings (public domain) and Hacker Public Radio (CC BY-SA 4.0). Not affiliated with SYSTRAN or OpenAI.