Most of our new posts were not in Google's index, and the cause was not content quality. Our internal links carried a ?lang=ko parameter, the URL pattern with that parameter was blocked in robots.txt, and Googlebot could not follow a single internal link. Once we switched default-language links to clean URLs without the parameter, the share of indexed pages rose from 57.6% (129 of 224, measured 2026-07-14) to 89.8% (238 of 265, measured 2026-08-16).
This post is the debugging path: how we stopped guessing, the four-bucket check that told us where to look, and the two commands that found the blocked links. It is also honest about where this diagnosis does not apply.
We blamed the content first, and it was the wrong call
At one point close to 85% of our new posts were not indexed. Publishing worked, and the sitemap listed every post. So we suspected the usual things: thin content, a young domain with little trust. We polished sentences, added images, and put summary cards on top of posts.
None of it moved the index rate. That was the clue we ignored for too long. If quality were the cause, indexing would be uneven: the posts we worked hardest on would get in. Instead, roughly the same share was missing everywhere, regardless of quality.
That gives you a rule of thumb. If non-indexed pages are scattered unevenly, suspect properties of individual pages (content, duplication, thin pages). If the whole site is blocked at a similar rate, suspect something every page shares: how links are generated, robots.txt, or server rendering.
Sort every URL into four buckets with the URL Inspection API
Instead of guessing, we ran the Search Console URL Inspection API over every published URL and sorted the answers into four buckets. The quota is 2,000 inspections per day, so a site with a few hundred posts can be checked in full. The core of the classifier looks like this (the service object is an authenticated Search Console API client):
import socket
socket.setdefaulttimeout(30) # one hung call should not stall the whole run
indexed, crawled_not, not_indexed, unknown = [], [], [], []
def classify(service, site_url, url):
resp = service.urlInspection().index().inspect(
body={"inspectionUrl": url, "siteUrl": site_url}).execute()
r = resp.get("inspectionResult", {}).get("indexStatusResult", {})
verdict = r.get("verdict", "")
cov = r.get("coverageState", "")
if verdict == "PASS" or cov == "Submitted and indexed":
indexed.append(url)
elif "crawled" in cov.lower() and "not indexed" in cov.lower():
crawled_not.append((url, cov))
elif cov:
not_indexed.append((url, cov))
else:
unknown.append((url, verdict or "?"))
The socket timeout matters. A single request that never returns will freeze a full run.
The buckets point at different causes:
- Indexed: nothing to fix.
- Crawled, not indexed: the crawler read the page and passed. The discovery path works, so look at content, duplication, and thin pages.
- Discovered, not indexed: Google knows the URL but has not read it yet. Think crawl budget or priority.
- Not discovered: the crawler does not know the URL exists. Look at the discovery path: internal links and the sitemap.
Our pages sat mostly in the last two buckets, which meant the problem was upstream of content.
How to tell a content problem from a crawl-path problem
Here is the procedure we would run again on any site, in order:
- List your published URLs, for example from your database.
- Check a small sample first. We ran 20 URLs before the full set to see the distribution.
- Decide the direction. Mostly "not discovered" or "discovered, not indexed" means go to step 4. Mostly "crawled, not indexed" means stop here and work on content.
- Read the blocking rules as they are served:
curl -s https://example.com/robots.txt
- Pull the internal links from the HTML a crawler actually receives, not from browser dev tools, and compare them against the blocked patterns:
curl -s "https://example.com/blog/post/some-slug" | grep -o 'href="[^"]*"' | sort -u | head -40
- If that output has no internal links at all, the problem is not blocking but missing server-side rendering. We fixed that part too and moved link rendering to the server.
- Fix link generation, deploy, and re-run the sample a few days later to compare distributions.
In our case step 5 showed the problem immediately. Every internal link had ?lang=ko appended automatically, and that URL shape matched a robots.txt disallow rule.
Why a blocked link pattern starves crawl budget
A crawler finds new pages mainly by following links from pages it already knows. When every internal link points at a disallowed URL, the crawler cannot follow any of them. The only remaining discovery route is the sitemap, and pages that are never discovered do not get crawl resources assigned. The entire site was bottlenecked on one link pattern.
We did not touch robots.txt. The rule was not wrong; our link generator was adding a parameter it did not need to. Loosening the rule would have let parameterized duplicate URLs into the index, trading one problem for another. The fix was to make default-language internal links point to clean URLs.
For more detail on the full write-up, including the original Korean walkthrough, see the full post on my-blog.org.
The result: 57.6% to 89.8% indexed
| Date | Indexed | Total | Index rate |
|---|---|---|---|
| 2026-07-14 | 129 | 224 | 57.6% |
| 2026-08-16 | 238 | 265 | 89.8% |
For URLs still not indexed after the fix, we queued them in a file and ran index requests separately. The lesson from automating that was that stop conditions mattered more than success handling. The script clicks only visible buttons, stops for the day after two quota messages in a row, stops on four consecutive errors or missing buttons because that suggests blocking, and removes processed URLs from the queue so the next day continues where it left off:
if quota_streak >= 2:
log("STOP: quota reached twice in a row, rest tomorrow"); break
if err_streak >= 4:
log("STOP: 4 failures in a row, suspect blocking"); break
In practice we saw the quota message after roughly ten requests a day.
Where this diagnosis does not help
- If your results pile up in "crawled, not indexed", Google is already reading your pages. Fixing links will not move the number. That needs a different fix, on the content and duplication side.
- On a very small site of a few dozen pages, the sitemap alone may be enough for discovery, so the same blocked pattern might show up as a smaller effect.
- An index request is a hint that asks for priority, not a guarantee. The search engine makes the final call.
- Numbers take time to move. Our two measurements are a month apart.
- This is a result from our own setup, which exposes a language parameter in URLs and has a matching
robots.txtrule. If your site does neither, the cause is probably elsewhere. The check, though, is cheap: extract the rendered links and compare them to your blocked patterns.
If you want to run this on your own site, the single most useful habit is to let the distribution of the four buckets tell you where to work before you rewrite anything. The longer version of this write-up, with the original scripts' context, is on my-blog.org.
Free tools I keep on my own site: 34 calculators and how-to guides - loans, severance pay, take-home salary, savings vs deposits. New ones go out in one email, no ads.


Top comments (0)