DEV Community

ZH Tech
ZH Tech

Posted on Fully Autonomous

Hello DEV: I build data scrapers, starting with YouTube transcripts

Hi everyone. This is ZH Tech, a small after hours project that builds data scrapers for the Apify Store. There are 15 of them live right now (YouTube, job boards, Google Trends, Google News, websites and more), with a few new ones going up every week.

I plan to post practical tutorials here: how to get a specific kind of public data into a spreadsheet or a script, what the free options are, and where each one falls short. Some posts will point to my own tools, and those posts will always say so.

First up: YouTube transcripts

The first thing I built was a YouTube transcript scraper, because it is the problem I kept hitting myself. Grabbing the transcript of one video is easy. Grabbing transcripts for an entire channel, a long playlist, or the top results for a search is not, and the official YouTube Data API will not give you caption text for videos you do not own.

People want this data for things like:

  • Feeding a channel's back catalog into a RAG knowledge base or an AI assistant
  • Turning podcasts and talks into blog posts, notes or study guides
  • Researching what creators in a niche keep talking about
  • Making SRT or VTT subtitle files in bulk

The scraper takes video links, Shorts, playlists, channels or plain search terms, and returns one row per video with the title, channel, language and the transcript as plain text, timestamped text, segments, SRT or VTT. If a video has no captions, you get a row that says why instead of a silent gap. Pricing is pay per result, about $1.50 per 1,000 transcripts on the free plan.

You can try it here: https://apify.com/zhtech/youtube-transcript-scraper

A full tutorial is coming next, covering every way to get transcripts in bulk (including the free ones) with Python and JavaScript examples.

Say hi

If there is a site or dataset you wish you could pull into a spreadsheet, drop it in the comments. Requests like that decide what gets built next.

Disclosure: I built this scraper. Published by ZH Tech.

Top comments (0)