How to Run Your First Technical SEO Audit with Screaming Frog, KWT Spider or Sitebulb

If you build or maintain websites, you eventually run into a familiar problem: the site looks fine in a browser, but something underneath is not working as expected. A page may return a 404. A redirect may point to another redirect. Important content may only appear after JavaScript runs. A canonical URL may be incorrect. Internal links may lead users through unnecessary hops. Or a page that should be indexable may accidentally contain a noindex directive. These problems are not always obvious from manually browsing a website. A technical SEO crawler can make them much easier to find. Tools such as Screaming Frog SEO Spider, KWT Spider, and Sitebulb can crawl websites and turn thousands of URLs into structured technical data. The interfaces and workflows differ, but the basic process is similar:
crawl → inspect → identify issues → verify → fix → crawl again
This guide walks through that process for your first technical SEO audit.

What Is a Technical SEO Audit?

A technical SEO audit looks at the elements of a website that affect how pages are discovered, crawled, rendered, understood and indexed.
The common areas are:

• HTTP status codes
• Dead links
• Redirects
• Canonical URLs
• robots.txt
• robots directives (noindex and others)
• XML sitemaps
• Page Titles and Meta Descriptions
• Heading style
• Site internal links
• JavaScript rendering
• Structured data
• Properties of images
• Page performance
• Mobile related issues

So the point is not to produce a long list of warnings. The objective is to determine what technical issues actually matter and what needs to be changed. That distinction is important when you start crawling bigger sites. A crawler can generate hundreds of findings, but you don’t need to fix all of them immediately.

Before You Start: Decide What You Are Auditing

Define the scope before starting a crawler. You are auditing:

• A whole production website?
• A staging environment?
• Specific subdirectory?
• List of URLs from migration?
• A JavaScript app?
• A little personal project?
• A large ecommerce site?

How you configure the crawl depends on the answer to this. For your first audit, start with a normal production website you have permission to crawl. You should also avoid sending an aggressive crawl at a server you do not control. Crawl speed, concurrency and resource usage are very important, especially on small hosting environments.

Step 1: Choose a Crawler

You can do the first workflow with some desktop SEO crawlers.

Screaming Frog SEO Spider Tool
Screaming Frog is a desktop SEO crawler that can crawl internal links, canonicals, JavaScript, images, CSS, external links, hreflang and other resources. Also includes documentation on redirect, canonical, JavaScript, structured-data and internal-link analysis.

KWT Spider
KWT Spider is a technical SEO desktop crawler for Windows, macOS and Linux, and is cross-platform. Current version includes Spider, List and Offline modes along with 29 diagnostic views covering such areas as technical SEO, metadata, links, images, structured data, performance and GEO/EEAT-related signals.
It has three crawl modes suitable for different development workflows:

• Spider – Crawl a live site beginning with a URL.
• List – Crawl a given list of URLs with no URL discovery.
• Offline – read a local folder without needing a web server.

Sitebulb
Sitebulb pairs website crawling with an audit-oriented interface. Current product documentation lists desktop and cloud versions, prioritized Hints, JavaScript crawling, internal-link analysis, structured-data checks, XML sitemap analysis and other technical SEO reports. Sitebulb also has its own HTML and Chrome versions of the crawler. Chrome crawler is meant for sites where you need to evaluate JavaScript rendering. You don't have to use all three. The main takeaway is to understand the audit process so that no matter which crawler you pick, you can use the same audit methodology.

Step 2: Start With a Small Crawl

If this is your first technical audit, don't immediately throw a massive website at the crawler. Start with a manageable site or a representative section. Enter the site's starting URL and begin a standard crawl.
For example:
https://example.com/
A basic crawl should give you enough information to start examining:

• URLs
• Status codes
• Page titles
• Meta descriptions
• Canonicals
• Indexability
• Internal links
• Images
• Redirects

Screaming Frog's getting-started documentation notes that its default configuration can be used for a basic website crawl, while additional configuration is available when a site requires more specialized analysis. Sitebulb similarly recommends starting with its standard settings before adjusting the audit configuration for specific requirements. For a first audit, that is good advice in general:
Don't configure everything before you understand what the crawler is already showing you.

Step 3: Check HTTP Status Codes

Begin with the basics. Every URL that the crawler encounters needs to have a response that is appropriate for what it is.
Search for:

• 200 — successful response
• 301 — permanent redirect
• 302 — temporal redirect
• 404 —not found
• 410 — out
• 5xx — server-side error

A 404 page is not necessarily a problem. For example, a deliberately removed URL can legitimately return a 404. The more useful question is: Are critical pages returning unexpected answers? Are internal links pointing users and crawlers to URLs that have been taken down?
This is where crawler reports are helpful. For example, Screaming Frog can find redirect chains, redirect loops, redirects to error pages and other response related issues.

Step 4: Identify Broken Internal Links

Then click on internal links.
Say this page:
/blog/technical-seo/

has a link to:
/resources/seo-guide/

but /resources/seo-guide/
now returns:

404 Not Found

A crawler can pinpoint the two sides of that problem:
Source page
↓
/blog/technical-seo/
↓
/resources/seo-guide/
↓
404
That gives the developer something to go on:

1. Remove the dead link.
2. Paste the correct URL.
3. Redirect the old URL if necessary.
4. Re-crawl the site.

Screaming Frog, in particular, tracks workflows for finding broken links and exporting the URLs and source pages that are affected.

Step 5: Review redirect chains

Redirects are fine. Long redirect chains are more troublesome.
For instance:
/old-page
↓ 301
/page-1
↓ 301
/page-final
↓ 200

If the original URL is still being linked to internally, you might want to consider updating the links to point directly to the final destination. These chains can be found at scale with a crawler. Screaming Frog's redirect reports can find redirect chains, loops and redirects that end up on error pages eventually. KWT Spider’s List mode can also be useful when the audit begins from a known set of URLs, such as URLs from a website migration or redirect mapping.

Step 6: Test Indexability

A key question to ask during a technical SEO audit is: Are the pages you want indexed actually indexable by search engines?
Search for:

• noindex
• nofollow
• robots.txt crawl restrictions
• URLs normalized
• non-200 responses
• contradictory orders

A good audit makes the difference between deliberate configuration and accidental configuration.
For example:
might be perfectly okay on an internal search-results page.
That same directive could be a serious problem on a site’s main product page. So don't assume every crawler warning is a bug. First, determine the purpose of the URL.

Step 7: Review canonical URLs

Another area where automated crawling can save considerable time is canonical URLs. A canonical would be:

During audit check for situations such as:

• Canonicals missing
• Canonical pointing to redirect
• Canonicals pointing to error pages
• Canonical chain
• Unexpected cross-domain canonicals
• Canonicals conflicting with the intended URL

Screaming Frog has dedicated canonical reports including canonical chains and non-indexable canonical targets. The crawler again checks the condition. But you still need to know if the implementation is intentional.

Step 8: Check Titles and Meta Descriptions

Now move on from infrastructure to page level SEO.
Watch for:

• Titles missing
• Titles repeated
• Extremely long titles
• Short titles
• Meta descriptions missing
• Duplicate descriptions

For example, a large ecommerce website may unintentionally generate:

For hundreds of products. A crawler uncovers that pattern right away. One of the most common audit workflows that has title and meta-description analysis included is the getting-started documentation from Screaming Frog. The next step is not necessarily to make every title unique because a crawler says “duplicate.” Have the pages truly represented different search intents?

Step 9. Check Internal Linking

Internal linking is particularly interesting to examine from a developer’s perspective, because it connects SEO to the site architecture.
Search for:

• Very few internal links to important pages
• Pages that are hard to access from the main site structure
• Linking to too many low-value URLs
• Broken internal links
• Redirects internal links
• Unexpected orphan pages

A crawler can see the structure of the site much faster than following links by hand. Screaming Frog offers site-architecture and internal-link analysis, while Sitebulb offers dedicated internal-link auditing and visualization features. For developers, this information can be a sign of deeper architectural issues at times. For example, if a significant part of the site is effectively hidden from the main navigation, it might be better to change templates or routing instead of editing a single page.

Step 10: Remember Javascript

This is where a basic HTML crawl may not tell the whole story.

A crawler that only examines the original HTML may not see everything that appears after rendering.
Screaming Frog provides a JavaScript rendering mode that uses Chromium to render pages and inspect the resulting HTML. Its documentation recommends this approach for JavaScript-heavy websites and frameworks such as React, Angular, and Vue.
Sitebulb similarly provides a Chrome crawler for JavaScript rendering.
When auditing a JavaScript application, compare:
Response HTML
↓
JavaScript execution
↓
Rendered HTML
Then ask:

• Is the main content present?
• Are important links present?
• Is the title correct?
• Are canonical tags correct?
• Are important resources loading?
• Is content dependent on client-side rendering?

This is one of the areas where developers and SEO specialists can work particularly well together.

Step 11: Review Structured Data

Structured data lets you describe your entities and the contents of your page in a machine-readable way. Depending on the website, you may see:

• Article
• Product
• Organization
• BreadcrumbList
• LocalBusiness
• FAQPage
• More Schema.org types

A crawler can assist you in locating where structured data exists and where potential implementation issues occur. For example, Sitebulb has a structured-data report and validator, and Screaming Frog has structured-data analysis built into its crawler. Don’t add structured data just because a crawler states a page doesn’t have it. First, verify that the markup is appropriate for the page.

Step 12: Inspect XML Sitemaps

Your crawler should also be compared with the XML site map of the site.
For instance,
Crawler found: 1,250 URLs
Sitemap contains 1,100 URLs
That difference is not necessarily an error. The crawler can find:

• Redirect URLs
• URLs of parameters
• URLs that are old
• Non-indexable pages
• URLs that should not be in the sitemap

The question of interest is:
Are the URLs that you really want search engines to find and index in the XML sitemap?
In Sitebulb’s technical audit workflow, you can crawl XML sitemaps as an alternative source of URLs to Google Analytics and Search Console.

Step 13: Track Performance and Resources

Technical SEO is more than just HTML. Modern websites rely on:

• Java Script
• Cascading Style Sheets (CSS)
• Pictures
• Font
• Video
• APIs.
• Third party resources

You can use a crawler to locate missing or problematic resources. For a more detailed performance analysis you may need additional tools such as browser-based performance testing or Lighthouse. Sitebulb’s crawler can pull in page-resource information and perform performance and mobile-friendly analysis through its Chrome crawler. Think of the crawler as one layer of investigation, not the only diagnostic tool you'll ever require.

Step 14: Make the Crawl a Developer Checklist

Don’t export everything and send a spreadsheet over to the development team when the crawl is complete. Convert findings into actionable tickets.
For instance:

Problem
Product pages have canonical URLs pointing to redirected URLs.
/product-a/
↓ Canonical
/products/product-a/
↓ 301
/products/a/
↓ 200

Suggested inquiry:
Look at the canonical generation logic and see if /products/a/ is the canonical destination.

Preference
High if affected URLs are important indexable pages. This format is far more useful to developers than:
"Correct canonical errors”

An Easy Audit Checklist

When you’ve crawled the first time, you can go through this checklist:

• [ ] Verify crawl scope
• [ ] HTTP status codes review
• [ ] Search for broken links inside
• [ ] Check redirect paths
• [ ] Look for redirect loops
• [ ] Indexability check
• [ ] robots.txt check
• [ ] Canonical URL
• [ ] Verify page titles
• [ ] Audit meta descriptions
• [ ] Browse titles
• [ ] Check Internal Linking
• [ ] Load JavaScript content
• [ ] Check the structured data
• [ ] Compare XML Sitemap URLs
• [ ] Check some key pictures/resources
• [ ] Review performance indicators
• [ ] Export important results
• [ ] Generate development tickets
• [ ] Debug
• [ ] Crawl again

You can use this checklist for future audits.

Screaming Frog vs. KWT Spider vs. Sitebulb: Which One to Choose

The three tools have overlapping functionality to crawl and analyze websites but they don't follow the same workflow.

• Screaming Frog SEO Spider:Detailed desktop crawling, configurable technical analysis, JavaScript rendering, internal-link and redirect investigation

• KWT Spider: A cross-platform desktop crawler with Spider, List, and Offline modes, local processing, and multiple diagnostic views

• Sitebulb: An audit-oriented workflow with prioritized Hints, visualizations, JavaScript crawling, and structured technical reports

These descriptions are based on the current documentation of each product and are not a ranking of the tools. Screaming Frog lists extensive crawl configuration and technical reports. KWT Spider describes its cross-platform desktop workflow and three crawl modes. Sitebulb details its audit workflow, prioritized Hints, visualizations and HTML/Chrome crawler options. Which one you choose in practice depends on the website and the person performing the audit. If you are a developer and your main need is to investigate HTTP responses, links, rendering, metadata, and architecture, start with those requirements. If you are an SEO specialist who does recurring audits, reporting and visualization may matter more. If you are auditing a migration, you may be more interested in a URL-list workflow than a full site crawl.

One Crawl Is Not the End

A technical SEO audit is most useful as a cycle:
Crawl
↓
Analyze
↓
Fix
↓
Re-Crawl
↓
Compare
The second crawl is often more useful than the first because it tells you whether the changes actually produced the expected result.

• Sitebulb, for example, includes audit comparison functionality for examining changes between audits.
• Screaming Frog also provides crawl comparison functionality as part of its desktop workflow.
• KWT Spider provides reusable crawl sessions that can be paused, saved, and resumed, according to its current product documentation.

Final Thoughts

Your first technical SEO audit doesn't need to be complicated. Start with the fundamentals:
status codes → links → redirects → indexability → canonicals → metadata → internal linking → JavaScript → structured data → sitemaps
Once you understand those areas, you can move into more advanced auditing. Screaming Frog, KWT Spider, and Sitebulb can all be used as entry points into this process, although each has its own interface, configuration options, and reporting workflow. For developers, the most useful mindset is to treat SEO findings as technical signals rather than a list of warnings to clear. A 404, noindex, redirect, missing canonical, or JavaScript-rendering difference is only the beginning of the investigation.
The real question is: What is the website doing, what should it be doing, and what code or configuration needs to change to get there? That's where a technical SEO crawl becomes useful—not just as an SEO report, but as another way to understand how a website actually works.