DEV Community

RapidBizz Team for RapidBizz

Posted on

The Unglamorous Maintenance Work That Keeps a Site Indexed

Nobody writes launch posts about maintenance. But in our experience running a production site, the work that protects your search visibility is mostly janitorial: watching crawl data, keeping structured data honest, and noticing small breakages before Google does. Here is the routine we run on our own site, including the two failures that taught us most of it.

Check Crawl Stats, not just coverage reports

A few months ago we opened Google Search Console's Crawl Stats report properly for the first time and found something uncomfortable: roughly 52% of Google's crawl requests to our domain were hitting dead URLs and returning 404s. The URLs were junk left over from an earlier compromise of the hosting account: probe paths under /api/, old feed and comment URLs, and random generated .shtml requests. Google was spending half its crawl visits on pages that did not exist, instead of on the pages we actually publish.

The fix was boring and effective. We cleaned out what did not belong on the server, then added explicit Disallow rules in robots.txt for the junk patterns (/api/, /wp-content/, /feed/, /comments/ and the *.shtml pattern), so crawlers stop wasting requests there. Crawl budget is not an abstract metric. On a small site it is the difference between Google re-crawling your service pages this week or next month. We now read Crawl Stats on a schedule: total requests, the 404 share, and average response time (ours sits around 370 ms, which told us the host was not the bottleneck).

Watch what the crawl is actually doing

Crawl Stats also breaks requests down by purpose, and that split taught us something. On our domain, refresh crawls of already-known URLs made up the overwhelming majority of visits, while discovery crawls, the ones that find new pages, were a small slice. That is normal, but it has a practical consequence: a new page with no internal links from pages Google already knows can wait a long time to be found. So part of our maintenance routine is link housekeeping. Every time we publish or substantially update a page, we add contextual links to it from related, already-indexed pages and check that older pages still point at the right targets after any merge or rename. Discovery is partly Google's decision, but you can make it an easy one.

Verify canonicals and sitemap health by hand

Once a month we fetch our sitemap and confirm three things: it returns 200, every URL it lists returns 200, and each page's canonical points at itself. This sounds trivial until you have watched a canonical problem sit unnoticed for weeks. Automated crawlers help, but a five-minute manual pass over the numbers catches drift that tools summarise away, like a sitemap still listing a page you merged last month.

Keep schema in sync with what the page actually says

Structured data drifts. Someone edits a visible FAQ answer, and the FAQPage JSON-LD keeps the old wording. We caught exactly this on one of our own pages recently: the visible answer carried a closing sentence with a guide link that the schema version of the same answer did not have. Google compares the two, and a mismatch is the kind of quiet inconsistency that undermines trust in the markup.

Our rule now: after any JSON-LD edit, parse every ld+json block on the file before it ships, and parse again from the live page after. String counts ("questions: 6") prove nothing. A parser proves it. We apply the same discipline to the newer surfaces too; our notes on how AI search engines read a site are in our guide to generative engine optimization for UAE businesses, and the same "visible text and markup must agree" principle runs through all of it.

The standing routine

The rest is a checklist, run whether or not anything looks broken:

  • Updates applied and verified on the live site, not just marked done in a dashboard
  • Backups taken before every edit, and a restore actually tested from time to time
  • Uptime and response checks, with someone accountable for reading them
  • A broken-link sweep across internal links after any page is added, moved or merged
  • Server log reviews for probe patterns and new junk URLs before they become a crawl problem
  • A monthly read of Search Console's manual actions, security issues and links reports

None of this is visible to visitors, and none of it trends on launch day. But indexing is earned continuously, and it is lost quietly, usually through neglect rather than through one big mistake.

This is the work bundled into our website maintenance services in Dubai: crawl hygiene, updates, backups, and the checks above, run on a schedule instead of after something breaks. If you maintain a production site yourself, steal the checklist. It has saved us more than once.

— RapidBizz Team (web, software and design agency serving the UAE and worldwide)

Top comments (0)