DEV Community

kouana
kouana

Posted on

Google Ignored Our Sitemap for Months. The Cause Was One Database Column

#wordpress #nextjs #webdev #seo

Our site had a problem we kept misdiagnosing: Google was slow to pick up changes. New pages sat in "Discovered – currently not indexed". Updated pages kept stale snippets for weeks.

Every obvious suspect was clean. The sitemap validated. robots.txt allowed everything. Every URL returned 200. So we blamed crawl budget and moved on. Twice.

Then we looked at the data inside the sitemap instead of its structure.

293 pages claimed to change at the same second

Our sitemap carried 402 URLs. Grouping them by lastmod:

curl -s https://example.com/sitemap.xml \
  | grep -o '<lastmod>[^<]*</lastmod>' \
  | sort | uniq -c | sort -rn | head
Enter fullscreen mode Exit fullscreen mode 
    293 <lastmod>2026-07-21T14:22:07+00:00</lastmod>
      6 <lastmod>2026-07-19T09:03:11+00:00</lastmod>
      4 <lastmod>2026-07-18T21:40:55+00:00</lastmod>
Enter fullscreen mode Exit fullscreen mode

293 of 402 pages shared one identical timestamp, down to the second. The whole file contained only 87 distinct dates.

The cause was mundane. Months earlier we ran a bulk update across the site. It rewrote post_modified on every row it touched. From then on our sitemap told Google that most of the site changed at the same instant and never changed again.

Put yourself on the crawler's side. A file claims three hundred unrelated pages were modified at 14:22:07 on the same day. That is not a signal, it is noise. The rational response is to stop trusting lastmod for that site and fall back to the existing crawl schedule. Which is what we were seeing.

Regenerating the sitemap fixes nothing

The bad value lives in the database, not in the file. Regeneration copies it faithfully.

Touching every post is worse. It stamps them all with a new identical timestamp and reproduces the problem with a fresher date.

What worked was deriving the date from the content itself. We compute a fingerprint of each page's rendered content and store the date that fingerprint last changed. That value, not post_modified, is what the sitemap publishes.

add_action('save_post', function ($post_id, $post) {
    if (wp_is_post_revision($post_id) || wp_is_post_autosave($post_id)) {
        return;
    }

    $fingerprint = md5($post->post_title . '|' . $post->post_content);
    $previous    = get_post_meta($post_id, '_content_fingerprint', true);

    if ($fingerprint === $previous) {
        return;                      // same content, so the date stands
    }

    update_post_meta($post_id, '_content_fingerprint', $fingerprint);
    update_post_meta($post_id, '_content_changed_at', current_time('mysql', true));
}, 10, 2);
Enter fullscreen mode Exit fullscreen mode

Then expose it through the REST layer the front end reads:

register_rest_field('page', 'sitemap_lastmod', [
    'get_callback' => function ($post) {
        $changed = get_post_meta($post['id'], '_content_changed_at', true);
        return $changed ?: $post['modified_gmt'];   // fallback for legacy content
    },
]);
Enter fullscreen mode Exit fullscreen mode

The property that matters is that it is honest by construction. A page whose content has not changed keeps its old date no matter how often you re-save it. A page whose content actually changed gets today's date. Nothing in the admin UI can inflate it.

The second half: your sitemap index is lying too

Most generators stamp index files with the generation time:

// ❌ every crawl sees a freshly modified index
lastModified: new Date(),
Enter fullscreen mode Exit fullscreen mode

That tells Google every index changed on every crawl, which is another way of saying your dates carry no information. Report the newest real date among the children instead:

const newestOf = (items: { lastmod: string }[]) =>
  items.reduce((max, i) => (i.lastmod > max ? i.lastmod : max), items[0].lastmod);

export default function sitemapIndex() {
  return SECTIONS.map((section) => ({
    url: `${BASE}/sitemap/${section.slug}.xml`,
    lastModified: newestOf(section.items),
  }));
}
Enter fullscreen mode Exit fullscreen mode

A trap we fell into ourselves while shipping this: we wrote newestOf, then forgot to use it in four of the entries. They kept emitting new Date(). Grep for your generator's date call after the change instead of trusting the diff.

Results

Metric Before After
Distinct lastmod values 87 337
Largest group sharing one timestamp 261 8
URLs in sitemap 402 402

Related: the pages that were never crawled at all

While investigating we checked the 41 URLs stuck in "Discovered – currently not indexed" and counted internal links pointing at each one. Eight had zero.

The correlation was exact. Every city page linked from its category hub was indexed. Every one omitted from that hub was stuck. Google found them in the sitemap, recorded them, then assigned them near-zero crawl priority because nothing on the site pointed at them.

We added the missing links to the hubs. Seven of nine pages got indexed with no manual indexing request at all. For comparison, manual requests in Search Console are capped at roughly ten per day. Internal links have no cap and fix the cause rather than the symptom.

What to check on your own site

  1. Export your sitemap and count distinct lastmod values. If that number is far below your URL count, your dates are noise and Google has probably stopped reading them.
  2. Check whether your sitemap index carries the generation timestamp rather than a real content date.
  3. Never run a bulk update that touches modification dates. The cost stays invisible for months, then it is your entire crawl behaviour.

The full write-up with the complete implementation is on our blog. It is in Arabic, but the code and the tables read the same in any language.

Top comments (0)