There is a difference between having the files and having a copy you can use.
Most people find this out the hard way: you save a page, open it a week later from
file://, and get a white screen, broken fonts, and links that point at a domain that
no longer resolves. The HTML is there. The site is not.
This post is about the second thing — making an offline copy that behaves like the
original, and knowing what you actually saved. It applies whether you use a crawler,
a browser extension, or wget.
Why "Save page as" is not enough
A modern page is not one file. It is a small graph:
- the HTML document
- CSS files, which themselves reference fonts and background images via
url()and@import - JavaScript files, sometimes loaded from a CDN
- images, some of which are lazy-loaded (
data-src,data-srcset, IntersectionObserver) - JSON for embedded data, SVG sprites, favicons, manifests
"Save page as → Webpage, Complete" gets some of this, but it flattens the directory
structure into a _files folder and rewrites links to point there. The result usually
works for one page and falls apart the moment you click anything.
What a usable offline copy needs
Five conditions, in order of how often they are skipped:
1. Links resolve from file://.
Absolute paths like /assets/app.css break the moment the document is not served from
a domain. They have to become relative paths, or the copy has to keep a directory
structure that mirrors the original.
2. CSS dependencies are followed, not just the stylesheet.
If main.css contains @import "print.css" or url(../fonts/inter.woff2), those are
separate requests. A crawler that only downloads <link rel="stylesheet"> hrefs will
produce a page with missing fonts and broken icons.
3. Lazy-loaded images are materialised.
The usual pattern stores the real URL in a data attribute and copies it into src when
the element scrolls into view. An offline copy that keeps the data attribute and drops
the JavaScript shows empty boxes. The fix is not to remove lazy loading from the copy;
it is to keep a small local script that performs the same "set src on scroll" step.
4. External and protocol-relative URLs are left alone.
//cdn.example.com/lib.js and https://fonts.googleapis.com/... are not part of the
site you are copying. Rewriting them to local paths produces a copy that silently
differs from the original, and it makes the diff useless later. Leave them external and
accept that those pieces need a network.
5. There is an index of what was saved.
This is the part everyone forgets. Two hundred files in a folder tell you nothing about
which page had which title, which URL it came from, or which pages failed. Without an
index, the archive is write-only.
Doing it by hand
You do not need a tool to apply the same rules:
1. Pick a scope: one section, a fixed depth, or a page list. Unbounded crawls of large
sites fail halfway and leave you with a mess.
2. Fetch the HTML. If the page only renders with JavaScript, this step cannot work -
see the limits below.
3. Parse the HTML and collect: <a href> (same origin only), <link rel=stylesheet>,
<script src>, <img src|data-src|srcset>, <source srcset>, CSS url() and @import.
4. Download into a mirrored directory path, then rewrite same-origin absolute paths to
relative ones. Keep external URLs untouched.
5. Record, for every file: original URL, page title, H1, meta description, canonical,
lang, content type, byte size, local path.
Step 5 is what turns a folder into an archive. A CSV with those columns opens in Excel or
Sheets and answers "what is this file, and where did it come from" without opening
anything.
url,title,h1,description,canonical,lang,kind,bytes,local_file
https://example.com/docs/,Docs - Example,Example API,Docs index,https://example.com/docs/,en,page,18422,example.com/docs/index.html
https://example.com/assets/app.css,,,,,en,asset,9123,example.com/assets/app.css
(Real exports usually also carry keywords and a kind column that distinguishes page
from asset, which makes filtering in Excel trivial.)
A flat CSV answers "what is this file". It does not answer "what shape is this site". The
same data as a hierarchy sheet does: TOP, Level 2 ... Level 10 columns with one row
per folder or page, parent cells merged down their subtree, the CSV columns to the right,
and the full URL as the last column so a row is still clickable. That is the layout people
already use in site-structure documents, and it takes about 150 lines of Go to write an
.xlsx by hand (a zip with five small XML parts) - no Office install, no library.
Two small details that save real time later:
- Write the CSV as UTF-8 with BOM if the audience uses Excel on Windows. Without the BOM, Japanese and accented text opens as mojibake.
- Write the map even if the crawl is stopped early. A partial map with a
status=partialnote is far more useful than nothing.
Where this breaks (say it before someone else does)
- Single-page apps. If the content only exists after client-side routing and API calls, a static copy cannot represent it. You can archive the shell, not the app.
- Logged-in areas. Anything behind auth is out of scope; a copier that logs in is storing someone's session, which is a bad idea.
- Infinite scroll. There is no "end" to detect. Use explicit page lists.
-
Sites you do not own.
robots.txtis the floor, not the ceiling. Permission comes first; the technical part is easy.
Checking the result
Before you trust an archive, do three checks:
- Open the entry page from
file://with the network disabled. If it looks right, links are correct. - Open two deep pages, not just the home page. Deep pages are where relative-path bugs live.
- Sort the CSV by
bytes. The largest files are usually videos or accidentally captured downloads; the zero-byte rows are the failures worth retrying.
The tool I ended up using
I do this often enough (client sites before a redesign, documentation I have permission
to keep) that I wrote a small Windows tool for it, SiteMirror. It follows the five
rules above: same-origin crawl with depth/page/asset limits, CSS url() and @import
following, a local lazy-load shim, untouched external URLs, and after every crawl it
writes the map as a filterable offline HTML tree, Markdown, the CSV described here, and
an Excel hierarchy sheet (one row per folder or page, parent cells merged, full URL in
the last column) that lists only pages which were actually saved - PDFs and downloads
are detected by their bytes and stay in the CSV instead of filling the tree.
It is a single .exe, no installer, no account, no telemetry, and the output is plain
files you can diff or archive. There is no free edition, so the honest advice is: try
wget --mirror first, and if the part you are missing is the index and the link
rewriting, that is what it solves. $9 one-time, no subscription.
Disclosure: I built the tool linked above. The rest of the post is the process I use
regardless of which tool does the fetching.
Top comments (0)