DEV Community

Cover image for My local LLM suggested 27 references. Only 7 passed verification.
Mario Vargas
Mario Vargas

Posted on Edited on

My local LLM suggested 27 references. Only 7 passed verification.

I record a lot of lectures and meetings, but I almost never watch them again.

What I actually want afterward is something I can read: a summary of what was discussed, the main topics, and any books or papers that were mentioned.

So I started building vtd (video-to-doc), an open-source CLI that turns audio and video recordings into structured Markdown or PDF documents.

At some point, I thought it would be useful to have the AI identify books and academic papers mentioned in a recording.

That sounded like a fairly straightforward feature.

Then I tested it with a real two-hour lecture.

My local model (qwen3.6:35b running through Ollama) suggested 27 references. I could only verify 7 of them against Open Library and Crossref.

That test led me to add a catalog verification step before a reference can appear in the final document.

What vtd does

The basic idea is simple. Give it a recording:

vtd lecture.mp4
Enter fullscreen mode Exit fullscreen mode

Without enabling AI, vtd uses whisper.cpp to transcribe the recording, splits it into sections based on pauses, extracts frames from the video, and generates a document combining everything.

It works locally, without an API key.

If you add --with-ai, a language model generates notes that include:

  • Summaries for each section
  • Main topics and themes
  • Follow-up tasks
  • A glossary of relevant terms

You can use a hosted model or a local one through Ollama.

I use Ollama to run the model locally.

Adding a verification step

I added a --references flag.

When enabled, the model identifies two types of works:

  1. References it believes were mentioned in the transcript.
  2. Additional works that might be useful as further reading.

For each one, it proposes an author, title, and publication year.

But instead of trusting that information, vtd searches two public catalogs:

  • Open Library, primarily for books.
  • Crossref, for academic publications and other indexed works.

To accept a reference, vtd checks that:

  • The title matches word for word, allowing spelling variations or a missing subtitle.
  • The first author's surname matches, including minor spelling or transliteration differences.

For example, Vygotsky and Vygotskiĭ can still match.

The final citation uses the catalog’s metadata.

To organize the bibliography, vtd checks the relevant transcript section for the author’s capitalized surname or a title of at least four words. It lists verified works with transcript evidence under Works referred to in the recording and other verified suggestions under Further reading.

Testing it with a public recording

I added a reproducible example to the repository.

I used President Kennedy's 1962 speech at Rice University, an 18-minute public-domain recording.

Here's the command:

vtd rice-speech.ogg \
  --language en \
  --whisper-model large-v3-turbo \
  --with-ai \
  --provider ollama \
  --ai-model qwen3.6:35b \
  --content-type lecture \
  --references \
  --pdf
Enter fullscreen mode Exit fullscreen mode

I ran it on a Mac with an M4 Pro and 48 GB of memory.

The results:

Step Time
Processing without AI 36 seconds
AI notes, references, and PDF 101 additional seconds

The Whisper model was already downloaded.

These timings are from a single run.

Bibliography

One of the eight proposed works matched a catalog record and appeared in the bibliography as further reading.

Works referred to in the recording

None detected.

Further reading (not mentioned in the recording)

Try it out

The project is open source and MIT-licensed:

GitHub: https://github.com/Mariotutal/video-to-doc

If you work with recorded lectures, meetings, or presentations, feel free to give it a try.

I’d be interested to hear how you use it with your own recordings. Share your experience in the comments or open an issue on GitHub.

Top comments (0)