Scott Williams

Posted on Nov 3 Edited on Nov 3

Download all images from a web page with Python (srcset, lazy loading and og:image included)

#beginners #python #tutorial #webscraping

"Download every image on this page" sounds like a one-liner: fetch the HTML, find every <img src>, download each one. On a real page that gets you a folder of small thumbnails, a handful of grey lazy-load placeholders, some tracking pixels, and possibly an HTML error page saved as photo.jpg. The large versions sit in srcset, the real URLs sit in data-src, and the best picture of the page is often only in a <meta property="og:image"> tag.

I wrote a small Python tool that handles those cases with the standard library only: fecklesspleb/page-image-downloader (MIT, one file, no browser). This post walks through what it has to deal with, then shows what came back from a real run.

1. Use Python's HTML parser, not a regex

html.parser.HTMLParser is in the standard library and copes with attribute order, quoting and entities (& in a URL query string is common on image CDNs). The tool subclasses it and records, in page order, every place an image can hide:

<img src>, plus data-src, data-lazy-src and data-original, which lazy-loading scripts use for the real URL while src holds a placeholder. When both exist, the lazy attribute wins.
<img srcset> and <picture><source srcset>, plus data-srcset / data-lazy-srcset.
<meta property="og:image"> and <meta name="twitter:image">, the image social sites show when the page is shared.
Optionally <a href> links ending in extensions you name, such as .pdf.

Every URL is resolved against <base href> (if there is one) and the page URL with urllib.parse.urljoin, #fragments are dropped, and each URL is kept once per page.

2. Parse srcset properly and take the largest

A srcset lists the same image at several sizes:

<img src="photo-400.jpg"
     srcset="photo-400.jpg 400w, photo-800.jpg 800w, photo-1600.jpg 1600w">

Splitting on commas breaks on CDN URLs that contain commas (/image/upload/w_800,q_80/photo.jpg). The HTML spec's rule is simpler and safe: a URL runs until whitespace, then the descriptor runs until the next comma. The tool parses it that way and keeps the candidate with the highest w, or the highest x density when there are no widths:

def best_srcset_url(value):
    best, best_key = None, None
    for url, desc in parse_srcset(value):
        key = (0, 1.0)                      # no descriptor means 1x
        m = re.match(r"^(\d+)w$", desc)
        if m:
            key = (1, float(m.group(1)))    # any width beats any density
        else:
            m = re.match(r"^(\d+(?:\.\d+)?)x$", desc)
            if m:
                key = (0, float(m.group(1)))
        if best_key is None or key > best_key:
            best, best_key = url, key
    return best

Wikipedia's thumbnails list a 2x version in srcset, so for the article's photos this means a 500 px file instead of the 250 px one in src.

3. Check the bytes, not the URL

A URL ending in .jpg can return a 404 page, a sign-in wall or a WebP. So the tool looks at the first bytes of every download: \x89PNG, \xff\xd8\xff (JPEG), GIF8, RIFF....WEBP, ftypavif, BM, the ICO header, or <svg. Anything else is recorded as not_an_image and not saved, and the file extension comes from the bytes, not the URL.

The same header gives the size without decoding anything. For a PNG the width and height are two big-endian integers right after IHDR:

if data.startswith(b"\x89PNG\r\n\x1a\n") and data[12:16] == b"IHDR":
    width, height = struct.unpack(">II", data[16:24])

JPEG takes a short loop over segments to the start-of-frame marker, and WebP has three variants (lossy, lossless, extended) with different layouts; the tool handles PNG, JPEG, GIF, WebP, AVIF, BMP and SVG. With the width known, --min-width 100 drops icons and spacers, and --min-kb 2 drops tracking pixels.

One trap: an SVG with width="34ex" has no pixel size. An early version of my tool fell back to the viewBox and reported a math formula as 5,664 px wide. Now any non-pixel unit leaves width and height empty, and --min-width keeps such files rather than guessing.

4. Skip duplicates by hash

The same logo often appears in the header and the footer under two URLs. Each file is hashed with SHA-256; a hash already saved in the run is recorded as duplicate, pointing to the first copy. The file name includes the first 10 hex characters of the hash (logo-3f9a0c12be.png), so two different files called logo.png never overwrite each other.

5. Be polite

urllib.robotparser checks robots.txt for every page and every image before it is requested, with a User-Agent that names the tool and links to the repo. A missing robots.txt (4xx) allows everything; one that errors (5xx) disallows the host, as RFC 9309 requires.
Requests to the same host are spaced 0.5 s apart by default.
HTTP 429 and 5xx are retried up to three times, honouring Retry-After; a 404 is not retried.

A real run

On November 3, 2026, I ran it against the python.org home page, the English Wikipedia article "Lighthouse", and one direct SVG URL on upload.wikimedia.org:

python3 page_image_downloader.py -i urls.txt -o images --min-kb 2 --min-width 100

Input	Candidates	Saved	Skipped
python.org home page	2	2	0
Wikipedia "Lighthouse"	41	27	7 narrower than 100 px, 4 under 2 KB, 3 blocked by robots.txt
Direct SVG	1	1	0

That is 30 files and 1,799,209 bytes in 19.5 seconds, with no errors.

A few things stood out:

python.org has two images in its HTML: the logo <img> and the og:image. I checked the raw HTML with grep and it has exactly one <img> tag. A page can look image-rich while its pictures come from CSS or scripts, which this tool (deliberately) doesn't run.
srcset did most of the work. 22 of the 30 saved files came from srcset. On Wikipedia, 40 <img> tags, 2 <source> tags and the og:image made 43 references, which came down to 41 unique URLs.
The share image was the biggest file. Wikipedia's og:image was a 960 × 1415 aerial photo (326,760 bytes); the article body only links a 500 px version of it. A plain <img> scraper would never see the large one.
robots.txt blocked three requests, all on en.wikipedia.org, which disallows /w/ and /wiki/Special:: two MediaWiki footer badges and a Special:CentralAutoLogin image. The tool never requested them.
The width filter mattered more than the size filter: 7 files were under 100 px wide, 4 were under 2 KB.

To check the manifest I opened all 25 raster files with Pillow: every width and height matched what the tool read from the headers, and every SHA-256 matched the file on disk.

A second run with --links pdf against the RFC 9309 info page on rfc-editor.org saved 4 files: the page's og:image, two logo SVGs, and the RFC as a 177,688-byte PDF. The tool checks that a "PDF" isn't really an HTML page (a sign-in wall, say) before it saves it.

What it doesn't do

No JavaScript. Images added by a script after load are not found. Server-rendered pages, lazy-load attributes and srcset are fine; for single-page apps you need a headless browser.
No CSS background images and no <video poster>.
One request at a time, which is fine for tens of pages.

Only download images you have the right to use, and keep the delay on.

Try it

git clone https://github.com/fecklesspleb/page-image-downloader
cd page-image-downloader
python3 page_image_downloader.py https://en.wikipedia.org/wiki/Lighthouse --min-width 100

You get images/en.wikipedia.org/ with the files and an images/manifest.csv listing every candidate with its status, size, width, height, alt text and hash. The 37 tests run offline against a local test server: python3 -m unittest discover -s tests -v.

The code, tests and the full list of statuses are in the repo: fecklesspleb/page-image-downloader.

AI disclosure: this article and the code were written with AI assistance. The numbers come from real runs on November 3, 2026, and I reviewed them before publishing. Not affiliated with the Python Software Foundation, Wikipedia, the Wikimedia Foundation or the RFC Editor.

DEV Community