Bulk download images from a URL list with Python (text file or CSV)

#tutorial #beginners #opensource #python

"Download these images" usually arrives as a spreadsheet or a text file, not as a web page to scrape. A loop around urllib gets you most of the way, then the real list shows up: a photo.jpg that is actually an HTML 404 page, the same logo under two CDNs, a host that answers 429, and filenames that collide on disk.

I wrote a small Python tool that takes that list seriously with the standard library only: fecklesspleb/bulk-image-downloader (MIT, one file, no browser). This post is about the problems it has to handle, not about hosting.

1. Read a text file or a CSV without drama

A text file is one URL per line, with # comments allowed. A CSV needs a clear column: url, image_url, link or src when present, otherwise the first column. Duplicate lines are kept once so you do not pay for the same request twice.

python3 bulk_image_downloader.py -i urls.txt -o images
python3 bulk_image_downloader.py -i urls.csv -o images --concurrency 8

2. Check the bytes, not the Content-Type alone

A URL ending in .jpg can return a sign-in wall or a soft 404 with Content-Type: text/html. The tool sniffs the first bytes for PNG, JPEG, GIF, WebP, AVIF, BMP, ICO and SVG. Anything else is recorded as not_an_image and not saved. The file extension comes from the sniff, so an application/octet-stream PNG still lands as .png.

Width and height are read from the header only, no image library required. That is enough to fill the manifest and to reason about what you actually got.

3. Skip duplicates by hash, name files safely

The same asset often appears under two URLs. Each body is hashed with SHA-256; a hash already saved in the run is recorded as duplicate and points at the first copy. The filename is <cleaned-stem>-<first 10 hex of sha256>.<ext>, so two different files that were both called logo.png never overwrite each other, and the stem is stripped of characters that are hostile to filesystems.

4. Retries, spacing and a concurrency limit

HTTP 429 and 5xx are retried (default three times), honouring Retry-After when it is numeric. A 404 is not retried. Requests to one host are spaced by a small delay, and --concurrency caps how many downloads run at once. That combination is what makes a 500-line CSV usable without melting a single origin.

5. Write a manifest you can audit

Every URL gets a row in images/manifest.csv:

status	meaning
`saved`	written under `images/<host>/...`
`duplicate`	same SHA-256 as an earlier file
`not_an_image`	bytes were not an image (often HTML)
`too_small` / `too_large`	failed `--min-kb` / `--max-mb`
`error`	bad URL, HTTP error, network failure

A JSON summary on stdout reports how many were saved and the byte total. The exit code is 1 only when nothing was saved and at least one request failed.

Try it

git clone https://github.com/fecklesspleb/bulk-image-downloader
cd bulk-image-downloader
printf '%s\n' \
  'https://upload.wikimedia.org/wikipedia/commons/c/c3/Python-logo-notext.svg' \
  'https://upload.wikimedia.org/wikipedia/commons/thumb/a/a7/Camponotus_flavomarginatus_ant.jpg/330px-Camponotus_flavomarginatus_ant.jpg' \
  > urls.txt
python3 bulk_image_downloader.py -i urls.txt -o images
python3 -m unittest discover -s tests -v

Only download images you have the right to use, and keep --delay / --concurrency polite on hosts you do not run.

The code, the offline tests (local http.server fixture) and the full status list are in the repo: fecklesspleb/bulk-image-downloader.

AI disclosure: this article and the code were written with AI assistance and reviewed before publishing. Not affiliated with the Wikimedia Foundation or the Python Software Foundation.

DEV Community