As developers, students, and lifelong learners, we consume an incredible amount of educational content via video. Whether it's a 3-hour university lecture on YouTube, a technical keynote from a conference, or a localized tutorial on Bilibili, video is a fantastic medium for explanation—but a terrible medium for reference.
Have you ever tried to find that one specific formula or code block in a two-hour video? You scrub back and forth through the timeline, squinting at compressed frames, wishing you just had a clean, written study guide with slide screenshots.
Doing this manually is a nightmare. You have to play, pause, take a screenshot, crop it, write down the spoken notes, organize them into a document, and repeat.
Today, we're diving into an advanced, open-source tool that automates this entire pipeline. By combining robust video utilities with state-of-the-art local AI processing, this project transforms any video link or local file into a structured, slide-illustrated Markdown and HTML document.
Let’s explore how it works under the hood and how you can run it entirely on your own hardware.
The Workflow: From Video URL to Clean Markdown
To understand why this tool is so powerful, let's look at the pipeline it orchestrates:
[ Video URL / File ]
│
▼ (yt-dlp)
[ Video & Audio Streams ]
│
├─────────────────────────────────┐
▼ (FFmpeg) ▼ (Whisper + CoreML/Vulkan)
[ Slide & Scene Extraction ] [ High-fidelity Transcript ]
│ │
└────────────────┬────────────────┘
▼ (Timestamp alignment)
[ Combined Slide & Text Segments ]
│
▼ (llama.cpp / OpenAI API)
[ LLM Restructuring & Polishing ]
│
▼
[ Beautiful Markdown & HTML Study Guide ]
By separating visual change detection from speech-to-text, and then reconciling them using an LLM, the tool creates a perfectly synchronized, textbook-like reading experience.
Under the Hood: The Modern Tech Stack
What makes this project stand out is its clever integration of high-performance local libraries. It doesn't just rely on expensive, cloud-hosted APIs; instead, it leverages the hardware acceleration built into your computer.
1. Media Ingestion with yt-dlp and FFmpeg
The pipeline begins by fetching the video. Thanks to yt-dlp, you can feed it URLs from virtually any platform (YouTube, Bilibili, Vimeo, or even direct streams).
Once the media is obtained, FFmpeg takes over to split the audio and video streams:
- Audio Extraction: Converts the audio to a standardized 16kHz mono WAV format, optimized for speech-to-text engines.
- Intelligent Slide Detection: Instead of taking screenshots at fixed intervals (e.g., every 10 seconds), the tool utilizes FFmpeg’s scene-change detection filters. It calculates the visual difference between consecutive frames to capture screenshots only when a slide transition actually occurs. This prevents duplicate images and ensures you capture every slide precisely when it changes.
2. High-Performance Local Transcription (Whisper + CoreML/Vulkan)
Transcription can be incredibly slow on standard CPUs. To make local processing viable, this project utilizes specialized bindings for OpenAI's Whisper model.
- On macOS: It leverages Apple Silicon’s Neural Engine via CoreML, enabling Whisper to run at blisteringly fast speeds while drawing minimal battery power.
- On Windows & Linux: It utilizes Vulkan-accelerated runtimes to offload the heavy tensor operations to your dedicated GPU (AMD, Intel, or NVIDIA), achieving performance that rivals cloud APIs.
The output is a highly accurate, timestamped transcription of the lecture.
3. Contextual Structuring with LLMs (llama.cpp or OpenAI)
Raw transcripts are hard to read. They contain filler words, grammatical slips, and lack formatting.
To bridge this gap, the tool pairs the visual "slide-change" timestamps with the corresponding transcribed text blocks. It then feeds these segments to an LLM.
You have two choices here:
-
Local Processing (
llama.cpp): Run open-source models like Llama 3 or Mistral directly on your machine. This ensures complete privacy and zero running costs. - Cloud Processing (OpenAI API): Use models like GPT-4o for maximum speed and synthesis quality.
The LLM is prompted to perform several key tasks:
- Clean up transcription noise (removing "umms", correcting technical jargon).
- Synthesize the text matching each screenshot into coherent, structured paragraphs.
- Format the content into clean Markdown headers, bullet points, and code blocks.
Step-by-Step: How to Use It
Ready to turn your lecture queue into a digital bookshelf? Here is how to set up and run the system on your machine.
Prerequisites
Ensure you have ffmpeg installed on your system.
-
macOS:
brew install ffmpeg -
Ubuntu/Debian:
sudo apt install ffmpeg
Installation
Clone the repository and install the Python dependencies:
git clone https://github.com/your-username/video-to-markdown.git
cd video-to-markdown
pip install -r requirements.txt
(Note: Depending on your hardware, you can install specialized wheels for whisper.cpp or llama-cpp-python to enable Vulkan/CoreML acceleration).
Generating Your First Study Guide
To process a YouTube lecture using your local GPU/Neural Engine and a local LLM, run:
python main.py \
--url "https://www.youtube.com/watch?v=dQw4w9WgXcQ" \
--engine local \
--model llama3 \
--output ./study_guides
If you prefer to use OpenAI's API for the text restructuring phase, simply export your API key and adjust the flags:
export OPENAI_API_KEY="your-api-key"
python main.py \
--url "https://www.youtube.com/watch?v=dQw4w9WgXcQ" \
--engine openai \
--model gpt-4o \
--output ./study_guides
The Output: Clean, Readable Markdown
The generated Markdown output is beautifully organized. It matches your screenshots with clean, synthesized text:
# Advanced System Design: Distributed Systems 101
## Section 1: Introduction to Latency vs. Throughput
*Timestamp: 01:15*
In this section, the lecturer introduces the core trade-offs in distributed system design.
While latency measures the time taken to complete a single transaction, throughput measures
the total number of transactions processed within a given timeframe.
### Key Takeaways:
* **Latency:** Goal is minimal delay (measured in milliseconds).
* **Throughput:** Goal is maximum data rate (measured in requests per second).
* Designing for high throughput often introduces queueing delays, which can degrade latency.
The tool also compiles this into a self-contained HTML report, allowing you to view your notes in any web browser with rich formatting and side-by-side layouts.
Why This Architecture Wins
- Uncompromising Privacy: For corporate training videos, proprietary research, or internal team meetings, you can keep the entire pipeline completely offline. Your data never leaves your machine.
-
Zero Run Costs: By relying on
llama.cppand local Whisper execution, you can process hundreds of hours of video without paying a single cent in API fees. - Multi-Platform Support: Whether you're running on an M-series MacBook, an AMD laptop, or an NVIDIA workstation, the Vulkan and CoreML backends ensure you get maximum hardware utilization.
Conclusion
This project shows what is possible when we combine traditional CLI media utilities with modern, local machine learning engines. By orchestrating yt-dlp, FFmpeg, Whisper, and Llama 3, we get a highly practical tool that solves a real-world productivity bottleneck.
Whether you're trying to study for exams, catch up on conference talks, or document internal presentations, this utility turns passive video watching into active, searchable knowledge bases.
Give the repository a star, set up the local pipeline, and start reclaiming your study time!
Have you experimented with local AI pipelines? What models are you running for text summarization? Let's discuss in the comments below!
Top comments (0)