ETag, Last-Modified and 304s: let Googlebot recrawl what actually changed

Every time Googlebot comes back to a page that hasn't changed, your server can send the whole thing again or answer 304 Not Modified with no body. The second option costs you almost nothing and leaves more of Google's crawl time for the pages that did change.

Google documents this in its overview of crawlers: its crawling infrastructure supports HTTP caching through ETag / If-None-Match and Last-Modified / If-Modified-Since. Its crawl budget guide, updated in July 2026, also lists "Support 304 (Not Modified) HTTP status codes" among its best practices. Most frameworks get this half right by default. This post shows how to get it fully right and how to prove it.

How the exchange works

First crawl:

GET /docs/pricing HTTP/1.1
Host: www.example.com

HTTP/1.1 200 OK
ETag: "a3f9c1"
Last-Modified: Wed, 14 Oct 2026 09:12:00 GMT
Content-Type: text/html; charset=utf-8

Next crawl, the crawler sends back what it saw:

GET /docs/pricing HTTP/1.1
If-None-Match: "a3f9c1"
If-Modified-Since: Wed, 14 Oct 2026 09:12:00 GMT

HTTP/1.1 304 Not Modified
ETag: "a3f9c1"

No body, so no download, no parsing and no rendering for that URL. If the page did change, you return 200 with the new body and a new ETag.

What Google recommends

Three points from the documentation are worth knowing before you touch any config:

1. If both headers are present, Google's crawlers use ETag, as the HTTP standard requires.
2. Google recommends ETag over Last-Modified for its crawlers, because ETags have no date formatting problems.
3. If you use Last-Modified, format the date per the HTTP standard. Google's example is Fri, 4 Sep 1998 19:15:56 GMT. It also suggests Cache-Control: max-age with the number of seconds you expect the content to stay unchanged, to help crawlers decide when to come back.

Google also notes that individual crawlers may or may not use caching depending on the product. Googlebot does when re-crawling for Search.

The bug most sites have

The usual failure isn't a missing header. It's an ETag that changes on every request even though the page didn't, so a 304 never happens.

Typical causes:

Cause	Symptom	Fix
Hash of the full HTML, which includes a CSRF token, nonce or timestamp	New ETag every request	Hash the content model (record ID + updated_at), not the rendered bytes
Several servers behind a load balancer, ETag built from inode or mtime	ETag differs per server	Use a content-based ETag
CDN or proxy compresses the response and rewrites the ETag	Strong ETag turns weak, or is dropped	Check what the edge sends, not what the origin sends
Last-Modified set to "now" by the framework	Every crawl looks like a change	Set it from the record's real modified time

Example: Express with a content-based ETag

Express creates a weak ETag from the response body by default. That breaks as soon as the body contains anything per-request. Generate it from data instead:

import crypto from "node:crypto";
import express from "express";

const app = express();
app.set("etag", false); // we set it ourselves

app.get("/docs/:slug", async (req, res, next) => {
  const page = await db.pages.findBySlug(req.params.slug);
  if (!page) return next();

  const etag = `"${crypto
    .createHash("sha1")
    .update(`${page.id}:${page.updatedAt.toISOString()}:${TEMPLATE_VERSION}`)
    .digest("hex")
    .slice(0, 16)}"`;

  res.set({
    ETag: etag,
    "Last-Modified": page.updatedAt.toUTCString(),
    "Cache-Control": "public, max-age=0, must-revalidate",
  });

  if (req.headers["if-none-match"] === etag) {
    return res.status(304).end();
  }
  res.send(render(page));
});

TEMPLATE_VERSION is a constant you bump when the layout changes, so a redesign still produces new ETags for every page.

toUTCString() gives Wed, 14 Oct 2026 09:12:00 GMT, the format Google asks for.

Example: static files on nginx

For static HTML, nginx already sends ETag and Last-Modified from the file's size and mtime. Two things break it:

location / {
    etag on;               # default, but make sure nothing turned it off
    if_modified_since exact;
    # gzip on; keeps a weak ETag (W/"...") since nginx 1.7.3, which is fine
}

• A build that rewrites every file on each deploy changes every mtime, so every page looks modified. Make your build skip unchanged outputs, or set mtimes from git (git restore-mtime from the git-tools package does this).
• Several servers with different deploy times give different ETags for the same file. Deploy the same artifact everywhere, or compute ETags at build time.

Test it with curl

URL=https://www.example.com/docs/pricing

ETAG=$(curl -sI "$URL" | awk -F': ' 'tolower($1)=="etag"{print $2}' | tr -d '\r')
echo "ETag: $ETAG"

curl -s -o /dev/null -w "%{http_code}\n" -H "If-None-Match: $ETAG" "$URL"
# expect 304

Run the first command twice. If the ETag differs between two requests a second apart, you have the per-request problem from the table above.

Test through your CDN hostname as well as the origin. The crawler sees the edge.

Check it in production

Two places tell you whether crawlers are actually getting 304s.

Your access logs. Count status codes for verified Googlebot requests:

grep "Googlebot" access.log | awk '{print $9}' | sort | uniq -c | sort -rn

User-agent strings can be faked, so for any decision that matters, verify the IP with a reverse DNS lookup, as Google's crawler verification guide describes.

Search Console Crawl Stats (Settings > Crawl stats). The "By response" breakdown shows the share of 304 (Not modified) responses. On a site where most pages rarely change, a share near zero usually means the ETag bug above.

What this does and doesn't do

Answering 304 doesn't make Google crawl more or index faster by itself. It makes each unchanged recrawl cheap for your server, which matters when the server is the bottleneck, and it gives Google a reliable signal that nothing changed. When you do change a page, the new ETag and a fresh lastmod in your sitemap tell the same story. For how to keep lastmod honest, see XML sitemaps that actually help indexing.

If a page has changed and Google still shows the old version weeks later, caching headers are rarely the cause. Look at whether the page is linked from anywhere Google crawls often.

---

Written by the IndexerPal team. We build tools for URL and backlink indexing. For the step after a page changes, see how IndexerPal works.