How to parse access logs, identify crawler requests, detect HTTP errors, and build a simple crawling diagnostics report with Python.
When working on technical SEO audits, I often need to answer a question that cannot be solved by looking at a website in a browser:
What are search engine crawlers actually requesting from the server?
A page may be listed in an XML sitemap but have no internal links. Another page may return HTTP 200 but contain almost no useful content. Some URLs may generate repeated server errors, while others may redirect through several intermediate addresses.
A regular website crawl is useful, but it does not show the complete history of requests made by search engine bots.
This is where server access logs become valuable.
In this tutorial, I'll build a small Python tool that analyzes Apache/Nginx access logs, extracts requests containing the Googlebot User-Agent, groups URLs by HTTP status code, and exports the results to CSV.
The goal is not to build a complete log analysis platform. I want a simple, reproducible tool that can help identify suspicious crawling patterns during a technical website audit.
1. What information can we get from access logs?
A typical Apache Combined Log Format entry looks like this:
66.249.66.1 - - [22/Nov/2026:08:15:10 +0300] "GET /catalog/ HTTP/1.1" 200 18452 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
It contains several useful fields:
| Field | Description |
|---|---|
| IP address | Client IP address |
| Timestamp | Date and time of the request |
| HTTP method | GET, POST, etc. |
| Request path | Requested URL path |
| Protocol | HTTP protocol version |
| Status code | Server response |
| Response size | Number of bytes sent |
| Referer | Referring URL |
| User-Agent | Client identification string |
For crawling diagnostics, I am particularly interested in:
- Which URLs were requested?
- How frequently were they requested?
- What HTTP status codes did they return?
- When was each URL requested last?
- Are there repeated server errors?
- Are there suspicious URLs with parameters?
- Are crawlers spending requests on unnecessary URL variations?
These questions help turn raw server logs into actionable technical information.
2. Important: User-Agent is not proof of Googlebot
Before writing the parser, there is one security detail worth mentioning.
It is easy to filter log entries containing the word Googlebot. However, User-Agent strings can be spoofed.
A request like this:
Mozilla/5.0 (compatible; Googlebot/2.1)
does not prove that the request came from Google.
For reliable identification, Google recommends verifying the requesting IP address using reverse DNS followed by forward DNS confirmation, or checking Google's published IP ranges.
Official documentation:
https://developers.google.com/search/docs/crawling-indexing/googlebot
In this first version, I will filter by User-Agent to demonstrate the parsing workflow. I will also show where IP verification should be added.
Important: The sample log entries in this tutorial are synthetic test data. They are not presented as verified Googlebot traffic.
3. Preparing the test environment
We need Python 3.10 or newer.
Create a project directory:
mkdir googlebot-log-analyzer
cd googlebot-log-analyzer
Create a file:
googlebot_analyzer.py
We will use only Python standard libraries:
-
refor parsing log lines; -
csvfor exporting reports; -
datetimefor timestamp processing; -
collectionsfor aggregating requests; -
argparsefor command-line arguments; -
urllib.parsefor handling URL paths and parameters.
No external packages are required.
4. Creating a sample access.log
Here are several example entries representing different crawling scenarios:
66.249.66.1 - - [22/Nov/2026:08:15:10 +0300] "GET /catalog/ HTTP/1.1" 200 18452 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.1 - - [22/Nov/2026:08:16:02 +0300] "GET /catalog/?page=2 HTTP/1.1" 200 19102 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.1 - - [22/Nov/2026:08:22:15 +0300] "GET /product/product-103/ HTTP/1.1" 301 312 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.1 - - [22/Nov/2026:08:25:40 +0300] "GET /product/product-104/ HTTP/1.1" 404 1620 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.1 - - [22/Nov/2026:08:27:12 +0300] "GET /product/product-105/ HTTP/1.1" 503 910 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
192.0.2.55 - - [22/Nov/2026:08:31:00 +0300] "GET /catalog/ HTTP/1.1" 200 18452 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
The last entry uses a documentation-only IP address. It is deliberately included to demonstrate why filtering by User-Agent alone is insufficient.
Save the complete sample as access.log.
5. Building the Python log analyzer
Now let's implement the parser.
The script will:
- Read access.log.
- Parse Combined Log Format entries.
- Filter requests containing Googlebot in the User-Agent.
- Group requests by URL.
- Count HTTP response categories.
- Record the last request timestamp.
- Export the results to CSV.
Here is the complete implementation:
import argparse
import csv
import re
from collections import defaultdict
from datetime import datetime
from urllib.parse import urlsplit
LOG_PATTERN = re.compile(
r'(?P<ip>\S+) \S+ \S+ '
r'\[(?P<time>.*?)\] '
r'"(?P<method>\S+) (?P<path>\S+) (?P<protocol>[^"]+)" '
r'(?P<status>\d{3}) (?P<size>\S+) '
r'"(?P<referer>[^"]*)" '
r'"(?P<ua>[^"]*)"'
)
def parse_timestamp(value):
return datetime.strptime(
value,
"%d/%b/%Y:%H:%M:%S %z"
)
def analyze_log(logfile):
stats = defaultdict(lambda: {
"requests": 0,
"2xx": 0,
"3xx": 0,
"4xx": 0,
"5xx": 0,
"last_seen": None,
"last_status": None
})
parsed_lines = 0
skipped_lines = 0
with open(
logfile,
"r",
encoding="utf-8",
errors="replace"
) as file:
for line in file:
match = LOG_PATTERN.match(line)
if not match:
skipped_lines += 1
continue
row = match.groupdict()
# This is only a User-Agent filter.
# It does NOT authenticate Googlebot.
if "Googlebot" not in row["ua"]:
continue
try:
request_time = parse_timestamp(row["time"])
except ValueError:
skipped_lines += 1
continue
parsed_url = urlsplit(row["path"])
url = parsed_url.path
if parsed_url.query:
url += "?" + parsed_url.query
status = int(row["status"])
item = stats[url]
item["requests"] += 1
if 200 <= status < 300:
item["2xx"] += 1
elif 300 <= status < 400:
item["3xx"] += 1
elif 400 <= status < 500:
item["4xx"] += 1
elif 500 <= status < 600:
item["5xx"] += 1
if (
item["last_seen"] is None
or request_time > item["last_seen"]
):
item["last_seen"] = request_time
item["last_status"] = status
parsed_lines += 1
return stats, parsed_lines, skipped_lines
def export_csv(stats, output_file):
with open(
output_file,
"w",
newline="",
encoding="utf-8-sig"
) as file:
writer = csv.writer(file)
writer.writerow([
"URL",
"Requests",
"2xx",
"3xx",
"4xx",
"5xx",
"Last Seen",
"Last Status"
])
sorted_items = sorted(
stats.items(),
key=lambda item: item[1]["requests"],
reverse=True
)
for url, data in sorted_items:
writer.writerow([
url,
data["requests"],
data["2xx"],
data["3xx"],
data["4xx"],
data["5xx"],
data["last_seen"].isoformat(),
data["last_status"]
])
def main():
parser = argparse.ArgumentParser(
description="Analyze crawler requests in access logs"
)
parser.add_argument(
"logfile",
help="Path to the access log"
)
parser.add_argument(
"--output",
default="googlebot_report.csv",
help="Output CSV filename"
)
args = parser.parse_args()
stats, parsed, skipped = analyze_log(args.logfile)
export_csv(stats, args.output)
print("Analysis completed")
print("Parsed crawler requests:", parsed)
print("Unique URLs:", len(stats))
print("Skipped lines:", skipped)
print("Report:", args.output)
if __name__ == "__main__":
main()
6. Running the analyzer
Run the following command:
python googlebot_analyzer.py access.log
The script will generate:
googlebot_report.csv
The CSV file can be opened in Excel, LibreOffice, or imported into another analysis tool.
For larger projects, I would also consider processing rotated log files and combining logs from multiple web servers.
7. What does the report tell us?
A simplified example of the resulting report:
| URL | Requests | 2xx | 3xx | 4xx | 5xx |
|---|---|---|---|---|---|
| /catalog/ | 1 | 1 | 0 | 0 | 0 |
| /catalog/?page=2 | 1 | 1 | 0 | 0 | 0 |
| /product/product-103/ | 2 | 1 | 1 | 0 | 0 |
| /product/product-104/ | 1 | 0 | 0 | 1 | 0 |
| /product/product-105/ | 2 | 1 | 0 | 0 | 1 |
Now we can identify several things worth investigating.
Product 103: redirect
The URL returned HTTP 301 and was requested again with HTTP 200 in our sample.
This is not necessarily a problem. However, if redirects form long chains, I would investigate whether the internal links point directly to the final destination.
Product 104: 404
The server returned HTTP 404.
If the product was permanently removed, this may be the correct response. If the product is still available, the routing or application logic needs investigation.
Product 105: 503 followed by 200
This is more interesting.
The server temporarily returned HTTP 503, but a later request succeeded.
I would check whether the issue is related to application availability, backend performance, rate limiting, or infrastructure instability.
A single 503 response does not prove a crawling problem. Repeated patterns and their timing are much more useful.
8. What I would add next
This script is intentionally small. It is a starting point, not a complete crawler analytics platform.
For a more advanced version, I would add:
- Reverse DNS and forward DNS verification of Googlebot IP addresses.
- IPv6 support.
- Sitemap parsing.
- Internal URL inventory from a website crawl.
- Detection of URLs missing from internal links.
- Redirect-chain analysis.
- Response-time analysis.
- Daily crawling trends.
- HTML and CSV reports.
- Filtering by Googlebot type.
The most useful next step would be combining three datasets:
- URLs listed in XML Sitemap.
- URLs discovered through internal links.
- URLs requested by verified Googlebot.
This makes it possible to identify pages that are listed in Sitemap but have no discovered internal links or no recorded crawler requests during the selected period.
That does not prove Google has never discovered them. It gives us a list of URLs that deserve further investigation.
9. Why I use server logs in technical SEO audits
I don't consider server logs a replacement for Google Search Console or a website crawler.
Each tool answers a different question.
| Tool | What it helps investigate |
|---|---|
| Screaming Frog | Internal link structure and URL discovery |
| XML Sitemap | URLs submitted to search engines |
| Server access logs | Actual HTTP requests received by the server |
| Google Search Console | Google's reported crawling and indexing information |
| Python scripts | Custom analysis and combining datasets |
Combining these sources provides a much better understanding of how a website is exposed to search engine crawlers.
If you're interested in the broader technical side of crawling architecture, I have also written about crawl budget and Googlebot behavior here:
Crawl Budget Waste: How Googlebot Spends a Website's Crawl Budget — SEO PROF 74
Conclusion
Server access logs are one of the most useful sources of information when investigating crawling problems.
They help answer questions that cannot always be answered through a browser or a standard SEO crawler:
- Which URLs did the bot request?
- What responses did the server return?
- Are there recurring HTTP errors?
- Are unnecessary URL variations consuming requests?
- Are important pages receiving crawler visits?
The Python script in this tutorial is deliberately simple, but it provides a useful foundation for more advanced crawling diagnostics.
My main takeaway: don't assume that a page is accessible to search engines just because it opens in your browser. Look at the actual requests, server responses, internal links, and indexing data.
That's where technical investigations become much more interesting.



Top comments (0)