How to search the Internet Archive from Python and get direct download links

I built this actor; it's a paid tool on Apify with a free trial credit.

The Internet Archive holds millions of free books, films, recordings and software, and it has public APIs. But turning "every Mark Twain audiobook, with MP3 links" into a clean table still means juggling two APIs, Solr query syntax, a 10,000-result paging limit and metadata that's sometimes a string and sometimes a list.

I'm an 18-year-old engineering student, and I wrapped all of that in an actor called Internet Archive Scraper. This post shows how to call it from Python and Node, what the data looks like, and one small project: a public-domain reading list with direct EPUB links.

How it works, briefly

The actor calls archive.org's own search API (advancedsearch.php) and, if you ask for files, the metadata API (/metadata/{identifier}). No browser, no login. Plain-word searches are limited to title, subjects and creator by default, because archive.org's full-text search is very noisy: when I sorted "moby dick" by downloads over full text, the top hit was a 1948 film noir. With the default, the top hits are actual Moby-Dick editions.

Python example

pip install apify-client

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")

run = client.actor("rel8ble/internet-archive-scraper").call(run_input={
    "queries": ["apollo 11"],
    "mediaTypes": ["movies"],
    "sortBy": "downloads",
    "maxResults": 20,
})

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["title"], item["year"], item["downloads"], item["url"])

Node example

npm install apify-client

import { ApifyClient } from "apify-client";

const client = new ApifyClient({ token: "<YOUR_APIFY_TOKEN>" });

const run = await client.actor("rel8ble/internet-archive-scraper").call({
  queries: ['creator:"Mark Twain"'],
  mediaTypes: ["audio"],
  sortBy: "downloads",
  maxResults: 50,
  includeFiles: true,
  fileFormats: ["mp3"],
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
for (const it of items) console.log(it.title, it.filesCount, "MP3 files");

What comes back

One item per archive.org item. This is a real row from a test run ("moby dick", sorted by downloads, file formats limited to PDF/EPUB), trimmed:

{
  "query": "moby dick",
  "identifier": "mobydickorwhale01melvuoft",
  "title": "Moby-Dick; or, The whale",
  "mediaType": "texts",
  "creator": "Melville, Herman, 1819-1891",
  "year": 1922,
  "publisher": "London Constable",
  "collections": ["robarts", "toronto", "university_of_toronto"],
  "downloads": 135030,
  "downloadsLastWeek": 678,
  "pageCount": 398,
  "url": "https://archive.org/details/mobydickorwhale01melvuoft",
  "filesCount": 2,
  "files": [
    {
      "name": "mobydickorwhale01melvuoft.epub",
      "format": "EPUB",
      "sizeBytes": 1433807,
      "url": "https://archive.org/download/mobydickorwhale01melvuoft/mobydickorwhale01melvuoft.epub"
    }
  ]
}

You also get subjects, language, license URL, rights, favorites, ratings, runtime for video and audio, ISBNs and a thumbnail URL. One detail I cared about: archive.org lists users' personal favourite lists as fav-<username> collections, and the actor strips those out.

Use case: a public-domain text corpus with direct links

The goal: the 50 most downloaded English books in archive.org's Project Gutenberg collection, with a direct link to each plain-text file, saved as CSV. That's a small, clean corpus for NLP experiments or a reading list.

import csv
from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")

run = client.actor("rel8ble/internet-archive-scraper").call(run_input={
    "collection": "gutenberg",
    "mediaTypes": ["texts"],
    "language": "eng",
    "sortBy": "downloads",
    "maxResults": 50,
    "includeFiles": True,
    "fileFormats": ["txt"],
})

with open("gutenberg-top50.csv", "w", newline="", encoding="utf-8") as f:
    w = csv.writer(f)
    w.writerow(["title", "creator", "downloads", "txt_url", "bytes"])
    for it in client.dataset(run["defaultDatasetId"]).iterate_items():
        txt = (it.get("files") or [{}])[0]
        w.writerow([it["title"], it.get("creator"), it["downloads"], txt.get("url", ""), txt.get("sizeBytes", "")])

In my test run, the top result was Amusements in Mathematics by Henry Ernest Dudeney (2,767,810 downloads), with its text at https://archive.org/download/amusementsinmath16713gut/16713-8.txt.

queries is empty here on purpose: when you only set filters, the actor lists the whole collection. Swap gutenberg for prelinger (old films) or oldtimeradio and change the format filter to h.264 or mp3. One thing I learned: Gutenberg items on archive.org carry text and HTML files, not EPUBs, and most have no year, so don't add a year filter to this one.

What it costs

Pricing is $1.00 per 1,000 results, and one result is one archive.org item, with or without its file list.

• The corpus above: 50 items = $0.05
• A 1,000-item catalog of a collection = $1.00
• Apify's free plan gives $5 of monthly credit, which covers about 5,000 items

Limits

These come straight from my testing:

• 10,000 results per query. archive.org's search API won't page further. Split by year or media type.
• File lists add one request per item. Audiobooks can have hundreds of files, so a 300-item run with files took about 5 minutes locally at concurrency 5. The default is now 10.
• Metadata is whatever the uploader entered. In a 300-item test, 78% had a date, 44% a license URL and 26% a publisher.
• It doesn't download files, it gives you the URLs. Check each item's license before reusing anything.
• One oddity I hit while building it: archive.org's newer "scrape" API cached responses and returned the same results for different queries. The actor uses the older, stable search API instead.

If something breaks or a field is wrong, the Issues tab on the actor page reaches me directly.

Internet Archive Scraper on Apify

---

This article was drafted with AI and published by me.