A month of correct on-site work sat completely unseen. Not because the work was wrong, but because we never checked what the crawler had actually decided to do with it.
A month of the right work, and nothing moved
We run our own automation for advai.cloud the same way we build it for clients, so when our own search position sat completely flat for a month despite real on-site improvements, we treated it as a bug to diagnose rather than a reason to work harder on the same thing.
The instinct when a number will not move is to assume the fix is more: more pages, more content, more links. We had that instinct too, and it would have been the wrong next step. Before adding anything, we checked what a search engine had actually done with the month of work already shipped. What we found was not a content problem. It was a visibility problem, and it was almost entirely our own doing.
The sitemap was telling the crawler to ignore almost everything
A sitemap carries a field called lastmod: the date a page last changed. It is the one field a search engine actually acts on. The other common fields, priority and how often a page changes, are widely ignored. Ours emitted a lastmod on 15 of 103 pages. The other 88 carried none at all.
No lastmod does not mean neutral. It means the crawler has no signal that anything changed, so it has no particular reason to look again. A month of real improvements had shipped across the site, and for the vast majority of pages, nothing in our own sitemap told anyone to come check.
The tempting fix, and why we did not take it
The easy patch is to stamp every page with the current date on every deploy. We did not do that, because it is dishonest in a way that gets punished. It asserts that every page changed today, every day, whether or not anything actually did. Search engines are specifically built to notice that pattern and discount it.
The honest fix took longer: derive each page's lastmod from the real date its content last changed, sourced from actual commit history rather than the moment the build happened to run. A page with no real history gets no date at all, rather than a guessed one. It is more work than one global timestamp. It is also the only version that tells the truth.
A second problem hiding behind the first: pages already declined
Fixing the sitemap surfaced a second, separate issue. A batch of our pages followed the same template with one variable swapped, the kind of page that is fast to produce because the same shape repeats. Checking each one's status individually showed that only a small fraction had ever been indexed. The rest had been crawled once and declined.
The pages that had been indexed shared a pattern too: each one was a specific, independent answer to a specific question. The refused pages were the same sentence structure with a place name changed twenty times over. We stopped building more of that template. Continuing would have meant adding volume to a pattern already measured, not assumed, to be one a search engine had decided against.
What actually moved, and what we are not claiming
We are not going to claim this one fix moved our ranking. Isolating the effect of a single change on search position honestly requires more time and more controls than we have run. What we can say plainly is the diagnosis: a month of correct work was sitting completely unread by the one system whose opinion decides whether it counts, and the reason was visible the moment we checked instead of assumed.
The habit we took from this is the part worth repeating to anyone running their own site. A crawler's view of your work is a measurable, checkable thing, not a background assumption. Shipping the work is not the same event as it being seen. And a formatting shortcut that quietly asserts something false, a build timestamp standing in for a real edit date, is exactly the kind of thing an automated system exists to notice.
Questions this post answers
Why would a site's search ranking stay flat after real SEO improvements ship?
Because shipping a change and a search engine seeing that change are two different events. If the sitemap does not signal that a page changed, the crawler has no reason to look again, and improvements can sit live and completely unread for as long as that signal is missing.
What does the lastmod field in a sitemap actually do?
It is the one field a search engine uses to decide whether a page is worth re-crawling. Priority and change-frequency fields are commonly ignored. A sitemap with no lastmod, or one covering only a fraction of a site's pages, gives the crawler nothing to act on for the rest.
Why not just stamp every page with the current date on every deploy?
Because that asserts freshness that never happened, on every single page, on every build. Search engines are built to notice exactly this pattern and discount it. The honest version derives the date from when a page's content actually last changed, which for us meant reading it from real commit history rather than the build clock.
What should you do if a search engine has stopped crawling some of your pages?
Find out which ones, and consider whether continuing to publish more of the same pattern is worth it. In our case, a set of templated pages had already been declined, and building more of that template would have added to a pattern already proven not to work rather than fixing anything.
Originally published at www.advai.cloud/blog/the-month-google-never-saw-our-site.
Top comments (0)