How the Wayback Machine Actually Works (I Went Down the Orkut Rabbit Hole)

It started with Orkut

I wanted to see the old Orkut login page. Google shut Orkut down years ago, so I figured it was gone forever. Then I typed orkut.com into the Wayback Machine and there it was: the logo, the layout, even the "Just like real life" text.

My brain immediately asked: wait, how is this even possible?

So I asked myself the most basic questions I could think of, and this post is me answering them. I got a few things wrong at first, and I'll point those out too.

Q1: What is the Internet Archive?

It's a non-profit digital library, founded in 1996 by Brewster Kahle. The goal is simple: free access to knowledge. It stores books, music, videos, software, and websites.

The Wayback Machine is its web archive. You type in a URL, pick a date, and see what that page looked like back then. A time machine for the web.

Q2: Is it just taking screenshots?

That's what I assumed, and it's wrong.

It saves the actual files the server sent: HTML, CSS, JavaScript, images. That's why you can still click around an old page and it behaves like a real website. A screenshot can't do that.

Q3: Does it take snapshots on a fixed schedule?

I thought it took 5 to 9 snapshots per month for every page. Also wrong.

There's no fixed schedule. A very popular site can be captured many times a day. A tiny personal site might be captured twice in five years. It mostly depends on how popular the page is and how many links point to it.

Q4: How does it even know a page exists?

The internet is huge, and nobody hands over a list of every page. So it discovers pages in a few ways:

1. Following links. A crawler starts from a known page, collects every link on it, visits those, and repeats. Popular sites get found constantly because the whole web links to them. A small site only gets found if something already known links to it.
2. Seed lists. The Archive imports URL lists from other sources, like Common Crawl and links cited on Wikipedia.
3. Save Page Now. Anyone can paste a URL into web.archive.org/save and it gets archived right away. Lots of small sites are in the archive only because one person submitted them.
4. Volunteer rescue crawls. A group called Archive Team runs big crawls when a service is about to shut down. Shutdowns like Orkut's attract that kind of attention.

So the honest answer for tiny pages: sometimes it doesn't know. If nobody links to a page and nobody submits it, it can be missed completely.

Q5: What is a crawler, and how does it visit a page?

A web crawler (or spider) is a program that browses the web automatically. The Internet Archive uses crawlers like Heritrix (its long-time workhorse) and Brozzler (which uses a real browser, so JavaScript-heavy pages load properly).

For one page, the crawler roughly does this:

1. Takes a URL from its queue.
2. Checks whether it's allowed to visit, such as robots.txt rules.
3. Sends an HTTP request, just like your browser would.
4. Gets the response: status code, headers, and body.
5. Saves everything.
6. Reads the HTML and pulls out links, images, scripts, and stylesheets.
7. Adds the new URLs to the queue.
8. Repeats, billions of times.

Q6: Where does all that data get stored?

Not as loose files. It's written into WARC files (Web ARChive format, ISO 28500). A WARC is just many records stored one after another. Each record has a header and then the content:

WARC/1.1
WARC-Type: response
WARC-Target-URI: http://www.orkut.com/
WARC-Date: 2014-09-01T10:22:31Z
WARC-Payload-Digest: sha1:ABC123...
Content-Type: application/http; msgtype=response

HTTP/1.1 200 OK
Content-Type: text/html
...

<html> ... </html>

The body is the full raw HTTP response, exactly as the server sent it. That's why pages can be replayed so faithfully.

Common record types:

Type	What it holds
warcinfo	Info about the crawl itself
request	The request the crawler sent
response	The server's full reply
resource	A standalone file
metadata	Extra info about the capture
revisit	"Same as an earlier capture," just a pointer

Q7: How does it handle billions of pages without exploding?

Two smart tricks:

• Per-record compression. Each record is gzipped on its own, so you can jump to a byte position and decompress just that one record instead of reading a multi-GB file.
• Deduplication. If an image hasn't changed since the last crawl, its hash matches, and the archive stores a tiny revisit record instead of another full copy.

Q8: How does it find one snapshot among billions?

That's the job of the CDX index. It maps:

URL + timestamp  →  WARC file + byte offset + length

When you open an Orkut snapshot:

1. You pick a date on the calendar.
2. The Wayback Machine looks up that URL and timestamp in the index.
3. It jumps to the exact spot in the right WARC file.
4. It decompresses that single record.
5. It rewrites the links, so images, CSS, and scripts load from the archive instead of the dead original site.
6. It serves the page to you.

Q9: What's the full timeline of one snapshot?

1. Discovery: the URL gets into the queue (via links, a seed list, or Save Page Now).
2. Crawl: the crawler requests the page and its assets.
3. Storage: responses are written into WARC files on Internet Archive servers.
4. Indexing: the capture is added to the CDX index by URL and timestamp.
5. Replay (maybe years later): you open the snapshot and the page is rebuilt from the stored records.

Q10: Then why doesn't the Orkut login work?

Because the archive saves a frozen frontend, not the app.

• No backend or database. It only captures what a browser receives. Orkut's profiles, scraps, and communities lived on Google's servers, which are gone.
• Nothing behind a login. The crawler can't sign in as you.
• Dynamic content is patchy. Big videos and content loaded by JavaScript after the page opens are often missed or only partly saved.
• Some sites are excluded. Some pages are never captured, or are removed on request.

So the page looks perfect, but the Sign in button leads nowhere.

Save your page

You can save page

web.archive.org/save 

add your page link

Wrap up

The Wayback Machine isn't magic. It's crawlers that follow links, WARC files that store raw responses, a CDX index that finds them again, and a replay layer that rewrites links. Once those four pieces click, the whole thing makes sense.

Have you ever found an old site of yours in the archive? Tell me in the comments .