A crawler reads a perfectly ordinary link: ../pricing. Your script appends it to the website address, requests the wrong page, and logs a failure. A second link begins with //, and the crawler quietly leaves the website it was supposed to inspect.
Both problems can hide behind a reassuring result: the HTTP request worked. What failed was the decision about which address to request.
This walkthrough builds a small resolver for a documentation crawler. It uses Python's standard library, needs no API account, and can be tested without visiting any website. The aim is specific: resolve ordinary links correctly, keep accepted links on one origin, and make rejected inputs visible.
Start with the page that actually supplied the link
Imagine a documentation page at https://docs.example.com/guides/install.html. It contains links to ../pricing, /faq, and ?lang=en. These are deliberately different inputs. Treating them all as fragments to concatenate hides those differences and makes debugging harder.
Use the current page address as the base. A website home page is often the wrong base for a link extracted from a nested document. Keep the raw extracted link in your diagnostic record so you can explain the eventual address.
Python provides urljoin for this operation. Its documentation also warns that an absolute second argument can replace the base address. Resolving an address and deciding whether to visit it are separate steps.
Run this small example before adding a network client:
from urllib.parse import urljoin
base = "https://docs.example.com/guides/install.html"
samples = [
"../pricing",
"/faq",
"?lang=en",
"//other.example.com/news",
]
for href in samples:
print(href, "->", urljoin(base, href))
The first result ends in /pricing; the second ends in /faq. The query-only input keeps the document path. The fourth input points at the other host. These outputs are reproducible local examples, not a claim about any real site's behavior.
That last case is useful. The resolver has correctly interpreted the input. Your crawler still needs a rule telling it that the resulting host is outside its assignment.
Define the boundary before writing the helper
For this example, an origin means the scheme, lowercase hostname, and effective port. The expected origin is supplied separately from the current page. This separation prevents a redirected page from silently becoming a new trusted base.
The chosen policy accepts HTTP and HTTPS addresses only. It refuses embedded usernames or passwords, backslashes, control characters, and leading or trailing whitespace. It removes fragments because this crawler collects documents rather than positions within documents.
These are application choices for a constrained documentation crawler. A general browser, a sitemap generator, and a login redirect handler can need different behavior. Keeping the choices explicit makes the helper easier to review.
The helper also checks the base itself. If the caller accidentally passes an external page, even a relative link is rejected. Otherwise a redirect could move the crawler's base, and a comparison against that moved base would approve the wrong destination.
This gives you an answer to a practical review question: what does a successful return value promise? It promises a resolved HTTP address within the configured origin, under the deliberately narrow input rules below. It does not promise that a request has succeeded or that a destination is safe to fetch.
Use a resolver with a visible rejection result
Save the following as resolver.py. A rejected input returns None, which lets the calling loop skip it without mistaking it for a request failure. If your application needs rejection reasons, replace that result with a small result object.
from urllib.parse import urljoin, urlsplit, urlunsplit
def origin(parts):
scheme = parts.scheme.lower()
if scheme not in ("http", "https"):
raise ValueError("unsupported scheme")
if not parts.hostname:
raise ValueError("missing hostname")
if parts.username is not None or parts.password is not None:
raise ValueError("credentials in URL")
port = parts.port
if port is None:
port = 443 if scheme == "https" else 80
return scheme, parts.hostname.lower(), port
def clean_input(value):
if not isinstance(value, str) or not value:
return False
if value != value.strip() or "\\" in value:
return False
return not any(ord(ch) < 32 or ord(ch) == 127 for ch in value)
def resolve_link(base, href, allowed_origin):
if not clean_input(base) or not clean_input(href):
return None
try:
base_parts = urlsplit(base)
if origin(base_parts) != allowed_origin:
return None
target = urlsplit(urljoin(base, href))
if origin(target) != allowed_origin:
return None
return urlunsplit((
target.scheme.lower(), target.netloc,
target.path, target.query, ""
))
except ValueError:
return None
The effective-port comparison avoids treating an explicitly written default port as a different origin. A genuinely different port is rejected. Accessing a malformed port can raise an exception; the helper returns the same rejection result instead of letting that input terminate the whole crawl.
Notice what the code does not normalize. It leaves the query intact. A page such as /search?q=python can differ from /search?q=sqlite. Removing every query parameter to reduce duplicates would be a separate content policy, with different consequences.
It also preserves the output authority spelling. The comparison uses a normalized hostname and effective port, but the returned address is not a complete canonical identity. If your queue requires stricter deduplication, define that normalization independently and test it against your own site's URLs.
Test the outputs that matter to the crawler
A useful test set contains accepted links and rejected links. Testing only ordinary relative paths would miss the boundary that motivated the helper. You can run the following as test_resolver.py, beside the helper.
import unittest
from resolver import resolve_link
BASE = "https://docs.example.com/guides/install.html"
ALLOWED = ("https", "docs.example.com", 443)
class ResolverTests(unittest.TestCase):
def test_expected_destinations(self):
cases = [
("../pricing", "https://docs.example.com/pricing"),
("/faq", "https://docs.example.com/faq"),
("?lang=en",
"https://docs.example.com/guides/install.html?lang=en"),
("#part",
"https://docs.example.com/guides/install.html"),
("https://docs.example.com:443/faq",
"https://docs.example.com:443/faq"),
]
for href, expected in cases:
with self.subTest(href=href):
self.assertEqual(
resolve_link(BASE, href, ALLOWED), expected
)
def test_rejected_destinations(self):
for href in [
"//other.example.com/news",
"https://docs.example.com.other.example/faq",
"http://docs.example.com/faq",
"https://docs.example.com:444/faq",
"https://user@docs.example.com/faq",
"https://docs.example.com:bad/faq",
"mailto:hello@example.com",
" /faq",
"/fa\tq",
"",
None,
]:
with self.subTest(href=href):
self.assertIsNone(resolve_link(BASE, href, ALLOWED))
def test_base_cannot_change_the_allowlist(self):
self.assertIsNone(resolve_link(
"https://other.example.com/page", "/faq", ALLOWED
))
if __name__ == "__main__":
unittest.main()
Run python -m unittest -v test_resolver. The suite contains three test methods covering seventeen input cases. All decisions happen locally; there is no DNS lookup, HTTP traffic, or dependency installation.
The near-match hostname is especially useful for catching a tempting shortcut: searching the complete address for a trusted hostname string. A string can contain the expected text and still describe a different host. Here the test asks for the exact parsed hostname.
When a site-specific case fails, keep the failing input and expected destination. That becomes a durable example for the next person who changes the helper, instead of a comment saying that URL handling is complicated.
Keep resolution out of the request loop
Place resolution between extraction and queueing. The extractor supplies the page address and raw link. The resolver supplies an accepted destination or a rejection. The request worker operates only on accepted destinations.
Record those outcomes separately. An out-of-scope link is a policy skip, while a timeout is a fetch failure. Combining them in one failure counter makes the crawler look unreliable and sends debugging effort toward the network.
For a first version, a compact record containing the base, input, accepted address, and outcome is enough. Avoid logging sensitive query values from private pages. A documentation crawler can often operate on public pages and retain only the examples needed for troubleshooting.
You can also deduplicate the accepted address before placing it in the queue. Fragment removal already collapses section links for this example, but it does not decide whether different queries represent the same document. Make that decision from the site's behavior, rather than a universal rule.
Handle redirects at the network boundary
The helper checks a requested address before fetching. It does not control redirects followed by the HTTP client. Python's urllib.request includes redirect handling, so the request path needs its own explicit policy.
For a constrained crawler, either disable automatic redirects and inspect each next destination, or use a client configuration that checks every redirect before following it. Checking the final response address is useful for diagnostics, but it happens after the request.
The same-origin helper is also not a complete SSRF defense. A service fetching user-supplied URLs needs network-level protections and a threat model covering DNS, private destinations, and redirect behavior. Do not promote this small crawler utility into a security boundary for arbitrary internet inputs.
Those requirements belong in the fetch layer. Keeping them there lets the resolver remain small and gives each layer a testable responsibility.
Apply the fix to one broken page first
Pick a page where your crawler currently produces a bad destination. Capture its actual base address and one extracted link. Reproduce the result locally, add the intended result to the tests, and only then resume the crawl.
This is a small change with an immediate observable outcome: the queue contains the addresses you meant to visit, and rejected links no longer masquerade as network failures. It gives a useful first repair before adding browser automation or a larger scraping framework.
Which input has caused the most confusing result in your crawler: a nested path, a query-only link, or a redirect? An exact base-and-link pair is more useful than a general description of a broken scraper.
Sources: Python URL parsing, Python HTTP request handling, and Python unittest.
Originally published on Dispatch.
Top comments (0)