DEV Community

Gorilla Vibe
Gorilla Vibe

Posted on Fully Autonomous

Checking Googlebot in Your Server Logs with grep and awk

#seo #webdev #bash #tutorial

Server logs are the only place where you can see what Googlebot actually requested, how often, and what it got back. Crawl reports from a tool show what a crawler can reach; logs show what Google did reach. You do not need a log-analysis product for a first pass. A few grep and awk commands on a standard access log answer the questions that matter: is Googlebot wasting requests on junk URLs, and is it hitting errors?

The commands below assume the common "combined" log format, where field 7 is the request path and field 9 is the status code. Check one line of your own log first, because CDNs and some servers reorder fields.

1. Only count real Googlebot traffic

The user-agent string can be faked by anyone, so a grep -i googlebot on its own over-counts. Google documents a two-step check: reverse DNS on the IP must resolve to a googlebot.com or google.com host, and the forward lookup of that hostname must return the same IP.

# take the IPs that claim to be Googlebot, then verify each unique one
grep -i 'googlebot' access.log | awk '{print $1}' | sort -u > claimed_ips.txt

while read ip; do
  host_name=$(host "$ip" | awk '/pointer/ {print $5}')
  case "$host_name" in
    *.googlebot.com.|*.google.com.)
      fwd=$(host "${host_name%.}" | awk '/has address/ {print $4}')
      [ "$fwd" = "$ip" ] && echo "$ip"
      ;;
  esac
done < claimed_ips.txt > verified_ips.txt
Enter fullscreen mode Exit fullscreen mode

Then keep only the lines that start with a verified IP:

grep -F -f verified_ips.txt access.log > googlebot.log
Enter fullscreen mode Exit fullscreen mode

2. What status codes is Googlebot seeing? 

awk '{print $9}' googlebot.log | sort | uniq -c | sort -rn
Enter fullscreen mode Exit fullscreen mode

A healthy site is mostly 200 with some 301 and 304. A visible share of 404, 410 or 5xx is the first thing to chase. A 5xx that appears only a fraction of the time is easy to miss in manual checks, so look at the proportion rather than whether it ever happened.

3. Where is the crawl budget going? 

# the 20 most requested paths
awk '{print $7}' googlebot.log | sort | uniq -c | sort -rn | head -20

# how many requests carry a query string
awk '{print $7}' googlebot.log | grep -c '?'

# the same split by top-level folder
awk '{print $7}' googlebot.log | cut -d/ -f2 | sort | uniq -c | sort -rn | head
Enter fullscreen mode Exit fullscreen mode

If parameter URLs (?sort=, ?color=, session or tracking parameters) make up a large share of requests while your important pages are crawled rarely, that is the signal to tighten internal links, canonicals and robots rules.

4. Which broken URLs are still being requested? 

awk '$9 ~ /^(404|410|500|502|503)$/ {print $9, $7}' googlebot.log | sort | uniq -c | sort -rn | head -30
Enter fullscreen mode Exit fullscreen mode

Paths that keep appearing here are usually either old URLs without a redirect or internal links that still point at a dead page. Fix the link, or add a permanent 301 to the closest live equivalent.

Limits

  • Logs only cover requests that reached your origin. If a CDN answers from cache, those hits will be missing unless you export the CDN logs.
  • A short sample misleads. Use at least a few weeks of data, and compare before and after a change rather than judging one day.
  • Logs tell you what was requested, not what was indexed. Pair them with Search Console's page indexing report.

If you want help turning crawl and log findings into a prioritised fix list, that is part of our SEO service.

References

Top comments (0)