DEV Community: Emir Baycan

Security Headers for SEO: HSTS, CSP, X-Frame-Options and More

Emir Baycan — Sun, 06 Sep 2026 15:05:32 +0000

Security headers for SEO are invisible to visitors but read by every browser. They harden a site against common attacks and signal that it is actively maintained. Each one is a single line of server config, and the audit flags the ones you are missing.

What a security header is

A security header is an instruction your server sends with every page, in the HTTP response, telling the browser how to treat the site: force HTTPS, block framing, refuse to guess file types, and so on.

Unlike content, headers are set once in your server or CDN configuration and then apply to every URL automatically. They do not change what a page looks like; they change what a browser is allowed to do with it. That makes them cheap to add and easy to forget, which is exactly why an audit checks for them. A missing header rarely breaks anything visibly, so it sits unnoticed until a security review, a penetration test, or a browser tightening its defaults surfaces it.

They are grouped under an SEO audit for two reasons. First, a hardened, well-configured site is a trust signal, and trust is increasingly part of how sites are judged. Second, one of these headers, HSTS, directly reinforces the HTTPS that Google already treats as a ranking factor.

The headers the audit checks, and what each does

Six checks, each with the one-line fix.

HSTS

Strict-Transport-Security tells the browser to only ever connect over HTTPS, closing the brief window where a first request could be intercepted on HTTP. Add Strict-Transport-Security: max-age=31536000; includeSubDomains.

Content-Security-Policy

CSP controls which sources of scripts, styles and images the browser will load, which is the primary defence against cross-site scripting. Start restrictive with Content-Security-Policy: default-src 'self' and widen it to the hosts you actually use.

X-Frame-Options

Stops other sites embedding your pages in an iframe, which prevents clickjacking. Add X-Frame-Options: SAMEORIGIN, or the modern equivalent frame-ancestors 'self' inside your CSP.

X-Content-Type-Options

Stops the browser guessing a file's type and running it as something it is not, a route to executing an upload as script. Add X-Content-Type-Options: nosniff.

Referrer-Policy

Controls how much of your URL is sent to sites your pages link to, so private paths and query strings do not leak. A sensible default is Referrer-Policy: strict-origin-when-cross-origin.

Secure cookies

Cookies without the Secure, HttpOnly and SameSite flags can be read over HTTP, stolen by scripts, or sent in cross-site requests. Set all three: Set-Cookie: id=...; Secure; HttpOnly; SameSite=Lax.

Adding them all at once

Most sites add every header in one place: the server or CDN config.

Because these are response headers, you do not edit page templates to add them. You set them once at the edge, in your web server config or your CDN, and they apply to every URL on the site. In nginx that is a block of add_header directives; on a CDN it is usually a response-header rule in the dashboard. Add the six above, then re-crawl, and the whole category clears at once.

One caution with Content-Security-Policy: it is the one header that can break a page if set too tightly, because it can block a script or stylesheet the page genuinely needs. Roll it out in Content-Security-Policy-Report-Only mode first, watch the browser console for what it would block, widen the policy to cover those legitimate sources, and only then switch it to enforcing.

Security headers in practice

Do missing headers actually hurt rankings?

Not directly, with one exception. Google does not rank a page lower purely because it lacks a Referrer-Policy or an X-Frame-Options header. The exception is HTTPS itself, which HSTS reinforces, and which Google has confirmed as a lightweight ranking signal.

The real value of the rest is defensive. A site that has been compromised, defaced, or used to serve malware will be flagged by Google Safe Browsing and can be dropped from results entirely, and these headers close the common routes to that outcome. They protect the rankings you already have rather than lifting you higher.

The one to add first

If you add only one header today, make it HSTS. It has the clearest search relevance because it locks in the HTTPS that Google already rewards, and it removes the small window on the first visit where a connection could be downgraded to HTTP and intercepted.

Add it only once you are confident every page and subdomain genuinely works over HTTPS, because HSTS is sticky: once a browser has seen the header, it refuses HTTP for that domain for the full max-age period. Test on a short max-age first, confirm nothing breaks, then raise it to a year.

Headers and mixed content are related

A Content-Security-Policy that only permits HTTPS sources doubles as a guard against mixed content, where an HTTPS page loads an asset over plain HTTP and the browser downgrades the whole page to "not fully secure".

Fixing mixed content at the source, by serving every asset over HTTPS, is the primary fix, but a strict CSP stops a future stray HTTP asset from ever reaching the browser. The two checks reinforce each other, which is why they sit in the same category.

See which headers your site is missing

Free to start. The audit lists the exact headers absent on your site and the pages affected.

Originally published at https://kalenux.com.tr/seo-guide/security/security-headers-guide

Image Alt Text for SEO: Missing, Too Long or Just the Filename

Emir Baycan — Sat, 05 Sep 2026 15:00:38 +0000

Alt text describes an image to people who cannot see it and to search engines that cannot interpret it. Missing, over-long, or filename-shaped alt text is a small fix with a real payoff in accessibility and image search. Here is how to get it right.

What alt text is for

The alt attribute is a short text description on an image. Screen readers read it aloud, browsers show it when an image fails to load, and search engines use it to understand what the image depicts. It serves accessibility and SEO at the same time.

A search engine cannot look at a photo and know what it shows the way a person can. It leans on the surrounding text and, heavily, on the alt attribute. Good alt text is how an image becomes eligible to rank in image search and how it adds relevant context to the page it sits on. But the larger reason to write it is accessibility: for a visually impaired visitor using a screen reader, the alt text is the image. A missing one is a blank where information should be.

The three findings in an audit map to the three common failures: no alt attribute at all, alt text so long it stops being a description, and alt text that is just the raw filename. The contrast below shows the difference:

Unhelpful, the filenameHelpful, a real description

The three alt-text failures, and their fixes

Each check the audit runs on images.

Missing alt attribute

The image has no alt at all, so screen readers announce nothing useful and search engines get no description. Fix by adding a concise alt describing what the image shows. The one exception is purely decorative images (borders, spacers), which should carry an empty alt="" so screen readers skip them.

Alt text too long

Alt over roughly 125 characters stops being a quick description and becomes a paragraph a screen reader has to read out in full. Fix by tightening it to the essential subject. If the image genuinely needs a long explanation, put that in the surrounding text or a caption, not the alt.

Alt is the filename

Values like DSC_0042.jpg or hero-image-final-v2 describe nothing. This usually happens when a CMS auto-fills alt from the filename on upload. Fix by replacing each with a real description of the image's content.

How to write alt text that works

Describe the image, in context, in a sentence.

Good alt text answers one question: if this image did not load, what would the reader need to know? Describe the subject and anything meaningful about it, in plain language, as if telling someone on the phone. "Red hiking boot on a rocky trail" beats both "boot" and "high-quality durable red waterproof hiking boot for mountain trails". Aim for a short, specific sentence, and let the natural description carry any keywords rather than forcing them.

Do not start with "image of" or "picture of", a screen reader already announces that it is an image. Do not stuff keywords; alt text overloaded with search terms reads badly to a screen-reader user and is exactly the pattern search engines discount. And for images that carry no information, decorative flourishes and spacers, use an empty alt="" so assistive technology passes over them instead of announcing a distracting filename.

Image alt text in practice

How much alt text helps rankings

For most pages the direct ranking effect is modest: alt text helps an image rank in Google Images and adds a little topical context to the page, but it will not move a competitive keyword on its own. Its weight grows for sites where images are the product, recipe photos, an e-commerce catalogue, a stock library, where image search is a real traffic source.

The accessibility case is stronger and unconditional. Every image that carries meaning needs alt text so that a visitor using a screen reader gets the same information as everyone else. Treating it as an accessibility requirement first, and an SEO bonus second, gets you both.

Alt text is not the filename, but the filename still matters

These are two separate signals. The alt attribute describes the image to people and engines; the filename is a smaller, secondary hint. A descriptive filename like latte-art.jpg is a minor plus over IMG_4821.jpg, but it never substitutes for real alt text.

The audit flags the filename-as-alt case specifically because it is the sign of an automated upload that filled both fields with the same useless string. Fixing the alt is the priority; renaming files is a nice-to-have you can do going forward rather than retrofitting across an entire library.

Fixing alt text at scale

On a large site, missing alt is rarely one image; it is a pattern from a template or a bulk import. The fastest wins come from finding the source: a product template that never outputs alt, a gallery component that drops it, an importer that copied filenames. Fix the template and hundreds of images gain alt at once.

For the long tail of one-off images, prioritize the ones that matter: hero images, product photos, anything a visitor would ask about. A decorative background does not need a hand-written description; a page's main illustration does. An audit that lists every affected image by page turns a vague "add alt text everywhere" into a finite, sortable task.

Find every image missing good alt text

Free to start. The audit lists images with missing, over-long, or filename alt, per page.

Originally published at https://kalenux.com.tr/seo-guide/content/image-alt-text-seo

Duplicate Content: Titles, Meta and How to Fix It

Emir Baycan — Fri, 04 Sep 2026 15:33:13 +0000

Duplicate content rarely earns a penalty, but it splits your ranking signals, hands Google the choice of which version to show, and wastes crawl budget on copies. Here's how it happens and how to consolidate it.

Is duplicate content bad for SEO?

The damage isn't a punishment, it's dilution. When the same content lives at several URLs, your ranking strength is divided among them instead of concentrated on one.

People hear "duplicate content penalty" and worry about the wrong thing. For ordinary technical duplication, there's no penalty, Google just picks one version to rank and ignores the others. The real cost is threefold: your link equity and ranking signals split across the duplicates instead of pooling on one strong page; you don't choose which version ranks (Google does, sometimes the wrong one); and crawlers waste budget fetching copies. Fix it not to avoid a penalty, but to concentrate your signals.

What causes duplicate content?

Most of it is technical and accidental, not copied text.

URL variations

The same page reachable with and without a trailing slash, over HTTP and HTTPS, with www and without, or with tracking and sort parameters appended. Each variant looks like a separate page with identical content. The audit flags URLs linked with and without a trailing slash, a common source.

Faceted navigation and pagination

Filter, sort, and parameter combinations can generate thousands of near-identical URLs. Paginated archives repeat much of the same boilerplate. These multiply duplicates fast and also burn crawl budget.

Duplicate titles, descriptions and H1s

The most common everyday duplication isn't whole pages, it's repeated title tags and meta descriptions and duplicate H1s across many pages, usually from a template that doesn't generate unique values. The audit flags duplicate titles, duplicate meta descriptions, and duplicate H1s directly.

How do you fix duplicate content?

The principle: one indexable URL per piece of content, with every signal pointing at it.

Choose the canonical version and add a rel=canonical tag on the duplicates pointing to it, so signals consolidate onto the master. Standardise URLs, pick one protocol, one host (www or not), and one trailing-slash convention, and 301-redirect the rest. Make real pages unique, give every genuine page a distinct title, meta description, and H1. Consolidate thin duplicates, merge near-identical pages into one stronger page, or noindex the ones that add no value (see thin content). Handle parameters, canonicalise parameter URLs to the clean version or manage low-value parameters in robots.txt. The result is one strong, clearly-signposted page instead of several weak, competing copies.

How do you check a site for duplicate content?

Most duplication is not plagiarism, it is your own site serving one page at several addresses. That makes it a crawl problem before it is a writing problem, and it means the check is mechanical rather than editorial.

Start with the URL variants of your own homepage

Before crawling anything, request these four addresses and watch what each returns. It takes a minute and it catches the single most common cause of site-wide duplication:

All four should end at exactly one URLhttp://example.com/http://www.example.com/https://example.com/https://www.example.com/

Three of them should 301-redirect to the fourth. If two or more return 200 independently, every page on your site exists at multiple addresses and your ranking signals are being split across all of them. The same test applies to trailing slashes, to uppercase variants of a path, and to /index.html against /. Each pair that answers 200 twice is a duplication source multiplied across your entire URL space.

Then find the parameter explosion

Crawl the site and group the resulting URLs by their path with all query strings stripped. Any path that appears many times with different parameters is a candidate. Tracking parameters, session identifiers, sort orders and filter combinations all produce URLs that serve identical or near-identical main content.

The distinction that matters is whether the parameter changes the main content in a way a searcher would want indexed separately. A filter that narrows a product list to one colour is usually not worth its own indexed page. A parameter that changes the page from one article to a different article obviously is. Sort orders almost never are: the same items in a different sequence is the same content.

Compare main content, not whole pages

Any similarity check that compares full HTML will report that every page on your site is roughly eighty percent identical, because every page carries the same header, navigation, sidebar and footer. That number is noise. A useful comparison strips the template first and compares only the main content region, which is what search engines weigh when deciding whether two pages are the same.

Once you are comparing the right thing, the results split cleanly. Near-identical main content across several URLs is real duplication to consolidate. Genuinely distinct main content that merely shares boilerplate is not a problem at all, no matter how high the whole-page similarity score looks.

Check the metadata separately

Titles, meta descriptions and H1s duplicate independently of body content, and they are worth their own pass because they are cheap to fix and visible in search results. Export every indexable URL with its title and description, then count occurrences of each. Anything appearing more than once is a page competing with itself for the same query, and a template that emits the same title for a whole section is a single fix that clears dozens of pages.

Pages whose main content is genuinely thin as well as duplicated need the treatment in the thin content guide rather than consolidation, because merging two weak pages usually produces one weak page rather than a strong one.

What is duplicate content in SEO?

Duplicate content means substantially the same main content reachable at more than one URL. The definition sounds obvious and the confusion is almost always about scope: what counts as "the same", and whether it means what people assume it means.

The definition, precisely

Search engines compare the main content of a page, not the whole document. Your header, navigation, sidebar and footer repeat on every page by design, and they are discounted accordingly. Two pages sharing eighty percent of their HTML because they share a template are not duplicates; two pages sharing their article body are, regardless of how different the surrounding chrome looks.

Substantially the same is deliberately loose. Identical text is the clear case, but so is a page that differs only in a swapped city name, a changed product colour, or a reordered list of the same items. The test is whether a searcher landing on the second page would get anything the first did not.

What it is not

There is no duplicate content penalty. This is the most persistent myth in the subject and it causes real harm, because it pushes people to delete or rewrite pages that were doing no damage. Google states plainly that duplicate content is not grounds for a manual action unless it is deceptive, such as scraping another site wholesale.

What actually happens is quieter. Search engines pick one URL from the duplicate set to show and filter the rest. The cost is not punishment, it is dilution: your ranking signals split across several addresses instead of accumulating on one, and you do not get to choose which version is kept. That is the real reason to consolidate.

Where it comes from on most sites

Rarely from copying. The usual sources are structural: a URL reachable with and without www or a trailing slash, a product listed under several category paths, a filtered or sorted view of a listing, a printer-friendly version, a session or tracking parameter appended to an otherwise identical page.

Each of those multiplies URLs without anyone writing a word twice, which is why duplication is a crawl-configuration problem before it is an editorial one, and why the checks in the previous section start with URL variants rather than with text comparison.

Judging and fixing duplication in practice

How much overlap counts as duplicate content?

There is no fixed percentage threshold. Search engines look at whether two pages are substantially the same in the parts that matter, which means the main content area rather than the furniture around it. Identical titles, descriptions and body content across several URLs clearly qualify as duplication.

Pages that share only boilerplate, the navigation, sidebar and footer that every page on a site carries, generally do not count, because their main content is genuinely distinct. The practical test is simple: strip away the template and ask whether what remains is meaningfully different from the other URL. If it is, you do not have a duplicate content problem on that page.

Internal duplication versus copies on other domains

Duplication within your own site is usually a technical consolidation problem, and it is solved with canonicals and redirects. Nothing about it is adversarial: you simply have several doors into the same room and need to nominate the main one.

Cross-site duplication, where the same content appears on other domains, raises a different question, namely which site is the original source. A canonical tag and clear original-publication signals help establish that. The case that carries real risk is deliberately copied or spun content, which can be treated as manipulative rather than as a technical accident.

Why canonical tags alone are not the whole fix

Canonicals solve consolidation, but they work best as part of a set. Pair them with consistent URLs, 301 redirects for the obvious variants, and unique titles, meta descriptions and H1s on the pages that are genuinely distinct.

The division of labour is worth remembering: canonical is the tool for "these are the same page", and uniqueness is the tool for "these should be different pages". Most sites need both at once, because most sites have some URLs that are true duplicates and others that are only duplicated because a template failed to generate distinct values for them.

Consolidate your duplicate pages and tags

Free to start. Use our free duplicate content checker to find duplicate titles, descriptions, H1s and URL variants across your site.

Originally published at https://kalenux.com.tr/seo-guide/content/duplicate-content-seo

XML Sitemap Best Practices

Emir Baycan — Thu, 03 Sep 2026 15:33:13 +0000

Your sitemap is a direct list of the URLs you want indexed. Keep it to clean, canonical, 200-status pages and it guides crawlers efficiently. Fill it with redirects and errors and it becomes noise. Here's how to do it right.

The one rule that drives every best practice

Your sitemap should list exactly the canonical, indexable, 200-status URLs you want in search, no more, no less. Everything below follows from that.

A sitemap doesn't directly boost rankings, it's a discovery aid that helps search engines find your pages and understand which URLs you consider canonical. That makes it most valuable for large sites, new sites without many backlinks, and deep pages. But its value depends entirely on cleanliness, a sitemap full of non-indexable URLs tells crawlers you want pages indexed that you've also told them not to index. The best practices below all serve the goal of a sitemap that matches your true indexable URL set.

The best practices

Each maps to a real check the audit runs against your sitemap.

Include only clean, indexable URLs

Every URL in the sitemap should be canonical, indexable, and return 200. Exclude redirects, 404s, noindex pages, and non-canonical URLs. The audit flags non-canonical URLs in the sitemap and 3XX (redirect) URLs in the sitemap: both contradict the sitemap's purpose.

Respect the size limits

Maximum 50,000 URLs or 50MB uncompressed per file. For larger sites, use a sitemap index that points to multiple child sitemaps, often split logically (by section, by content type). The audit flags a sitemap that's too large.

Keep lastmod accurate

Provide a truthful lastmod for each URL so crawlers can prioritise genuinely updated pages. Don't stamp every URL with today's date and never use a future date, the audit flags missing lastmod and future-dated lastmod, both of which train crawlers to ignore the field.

Declare it and keep it valid

Reference the sitemap in robots.txt with a Sitemap: line and submit it in Search Console. Make sure it's valid XML and loads quickly, the audit flags sitemap syntax errors and sitemap timeouts. And don't list the same URL in multiple sitemaps; the audit flags pages appearing in multiple sitemaps.

Keep the sitemap and your indexable set in sync

The subtle failure isn't a bad URL in the sitemap, it's a good URL missing from it.

Two sets should match, the URLs in your sitemap, and the canonical indexable URLs on your site. When a non-indexable URL sneaks into the sitemap, you send a contradictory signal. When an indexable page is missing from the sitemap, it may be crawled less and discovered later, the audit flags indexable pages not in the sitemap for exactly this reason. It also flags pages dropped from the sitemap since the last crawl, which often signals an accidental removal. Treat the sitemap as a generated reflection of your indexable pages, not a hand-maintained list that drifts out of date.

Common sitemap questions settled

Sitemaps on small sites

A sitemap is still recommended for a small site. It is a low-effort way to ensure discovery and to signal which URLs you consider canonical, and most platforms generate one for you.

That said, a small, well-linked site will mostly be crawled fine without one, because internal links alone are enough for crawlers to reach every page. The value of a sitemap grows with site size and structural depth, so treat it as insurance early on and as genuine infrastructure once the site is large.

Images and video in the sitemap

You can add image and video extensions to a sitemap to aid their discovery, which helps when media is a significant part of what your site offers, such as a product catalogue or a video library.

The core principle still applies: only include media tied to indexable, canonical pages. Listing assets that sit on noindex or non-canonical pages reintroduces the same contradiction the rest of these practices exist to avoid.

How often should the sitemap update?

The sitemap should regenerate whenever pages are added, removed, or change canonical or index status, so it always reflects the current indexable set rather than a past one.

Most platforms generate it dynamically, which makes this automatic. The key is that it stays accurate rather than becoming a stale snapshot: a sitemap regenerated on a schedule that ignores content changes will drift out of sync with the site exactly when the site is changing most.

Audit your sitemap against your real pages

Free to start. Find redirects, errors, non-canonical URLs and missing pages in your sitemap.

Originally published at https://kalenux.com.tr/seo-guide/sitemaps/xml-sitemap-best-practices

Redirect Chains & Loops: Why They Leak SEO Authority

Emir Baycan — Wed, 02 Sep 2026 15:00:27 +0000

A clean redirect passes nearly all your ranking authority to the new URL. Chain several together and you add latency, waste crawl budget, and risk leaking equity. Loop them and the page breaks entirely. Here's how to find and collapse them.

Why chains and loops happen

Redirects accumulate over a site's life, every migration, rename, and protocol change adds a rule, and the rules start tripping over each other.

No one sets out to build a redirect chain. They emerge: you move a page, then move it again a year later, and now the original URL hops twice. You add an HTTPS redirect, then a trailing-slash redirect, and a URL passes through both. Over a few years you get a tangle of multi-hop redirects, and occasionally two rules that point back at each other and form a loop. Here's the difference, and the fix:

Chain (bad): three requests to reach the page/old-post 301 → /blog/old-post/blog/old-post 301 → /blog/new-post/blog/new-post 301 → /articles/new-post # finalCollapsed (good): every old URL points straight to the final/old-post 301 → /articles/new-post/blog/old-post 301 → /articles/new-post/blog/new-post 301 → /articles/new-post

The audit flags both chains and loops because both cost you: chains add latency and risk equity loss at each hop, and a loop never resolves at all, taking the page completely offline for users and crawlers.

What chains and loops cost you

One degrades performance and authority; the other breaks the page completely.

Chains: latency, crawl waste, and equity risk

Every hop in a chain is an extra round-trip for the user (slower page loads) and an extra request for the crawler (wasted crawl budget). While Google follows several hops and consolidates signals, each hop adds risk, a chance of equity loss, a chance the chain breaks if one hop later 404s, and slower indexing of the final page. There is never a benefit to a chain: only downside to minimise.

Loops: the page becomes unreachable

A loop never resolves. The browser eventually shows a "too many redirects" error and the crawler abandons the URL. The page is effectively offline: no users, no indexing, no rankings. Loops are almost always caused by conflicting rules (an HTTPS rule and a trailing-slash rule each undoing the other) and are urgent, they take a live page completely out of search.

How to fix chains and loops

Collapse, then point links at the final URL.

Step 1, Map the hops. Crawl the site and identify every redirect and where it ultimately lands. The audit surfaces redirect chains and the pages whose internal links point to a redirect. Step 2, Collapse each chain. Rewrite the redirect rules so the first URL goes straight to the final destination: turn A→B→C into A→C and B→C. Step 3, Break loops at the source. Find the conflicting rules and fix the order or logic so the cycle resolves to a single final URL. Step 4, Update internal links. Change internal links so they point at the final URL directly, not at a redirecting one, this removes the hop entirely for crawlers and users. Step 5, Use the right status code. Permanent moves should be 301s; see 301 vs 302.

Chains, links and loops in detail

How many hops are acceptable

Aim for zero unnecessary hops: every redirect should go straight to the final URL. There is no hop count that is actively good, only a number you have not got around to removing.

Google follows several hops in practice, so a two-step chain is unlikely to be catastrophic, but anything more than one is worth collapsing, and long chains risk being abandoned before the crawler reaches the destination. The simplest rule to work to: one redirect, directly to the destination.

Internal links pointing at redirects

Internal links should never point at a URL that redirects. If you know the final URL, link to it directly.

Linking to a redirecting URL wastes a hop on every crawl and every click, adding latency for users and requests for crawlers, and it is entirely avoidable since you control both ends of the link. The audit flags internal links pointing to redirects for exactly this reason: the redirect rule can stay as a safety net for external links and bookmarks, but your own pages should bypass it.

What causes most redirect loops

Redirect loops are almost always caused by conflicting server rules, most often an HTTPS-enforcement rule and a trailing-slash rule, or a www against non-www rule, that each rewrite the URL in a way that re-triggers the other.

Each rule is correct in isolation, which is why loops survive review: the fault only appears in the interaction. Fixing the order and conditions of those rules, so that one runs to completion before the other applies, resolves the loop.

Collapse the chains leaking your authority

Free to start. Find redirect chains, loops and links pointing to redirects across your site.

Originally published at https://kalenux.com.tr/seo-guide/redirects/redirect-chains-seo

robots.txt for SEO: How Not to Block Your Own Site

Emir Baycan — Tue, 01 Sep 2026 15:00:37 +0000

robots.txt controls what search engines crawl, and a single over-broad rule can hide your whole site. Here's how it actually behaves, why blocking isn't the same as deindexing, and how not to block your own pages.

What robots.txt is for

It's a crawl-management tool, not an indexing tool. That one distinction explains almost every robots.txt mistake.

robots.txt is a plain text file at yourdomain.com/robots.txt that crawlers read before crawling. With User-agent, Disallow and Allow rules you tell bots which paths to skip, and you declare your sitemap. A sensible file looks like this:

A healthy robots.txt# applies to all crawlersUser-agent: Disallow: /admin/Disallow: /cart/Disallow: /?sort= # block faceted/sort paramsAllow: /Sitemap: https://example.com/sitemap.xml

You'd typically block admin areas, internal search, login/cart pages, and infinite parameter URLs that would waste crawl budget. What robots.txt does not do is remove pages from the index, and conflating those two jobs is where most sites get into trouble.

Blocking crawling vs blocking indexing

The most important thing to understand about robots.txt.

Disallow stops crawling, not indexing

If you Disallow a URL, crawlers won't fetch its content, but Google can still index the URL if it's linked from elsewhere, showing it in results without a description ("No information is available for this page"). So disallowing a page you wanted hidden can leave a bare, description-less listing in search.

The noindex trap

Here's the trap that catches people: if you want a page out of search, you might add a noindex tagand Disallow it in robots.txt to be thorough. But because robots.txt blocks crawling, Google never fetches the page, so it never sees the noindex, and the page stays indexed. To deindex a page, you must allow crawling so Google can read the noindex. Disallow and noindex work against each other.

The mistakes that block your own site

Real robots.txt checks the audit runs, and how each goes wrong.

Disallow: / from staging. Staging sites often block all crawlers. If that file ships to production, your entire site becomes uncrawlable. This is the robots.txt equivalent of the stray noindex. Blocking indexable pages. An over-broad path rule can cover pages you want ranked, the audit flags robots.txt blocking indexable pages. Blocking the sitemap. A rule that disallows your sitemap path stops crawlers reading it. No sitemap declared. robots.txt should include a Sitemap: line pointing to your XML sitemap. Blocking CSS/JS. Google renders pages; blocking the assets it needs to render can hurt how it sees the page. Missing robots.txt entirely isn't fatal, but a present, correct file is best practice, the audit flags a missing robots.txt too.

Placement, blocking and sitemap declaration

Where the file belongs

robots.txt lives at the root of the domain, at https://yourdomain.com/robots.txt, and nowhere else. A crawler will not look for it in a subdirectory. The rules it contains apply only to the host that served it, so every subdomain needs its own file: a robots.txt on the main domain says nothing about shop.yourdomain.com. The file must return a 200 status and be served as plain text. A robots.txt that 404s is treated as "crawl everything"; one that returns a 500 can cause Google to pause crawling the site altogether.

How do you block a page from Google properly?

To keep a page out of search results, allow it to be crawled and add a noindex meta tag or an X-Robots-Tag header. This is the opposite of most people's instinct, but it is the only combination that works: Google has to fetch the page to see the noindex directive, and a robots.txt Disallow prevents exactly that fetch. Blocking the URL instead leaves it eligible to appear in results, stripped of its description.

For anything genuinely sensitive, use authentication rather than either mechanism. Disallowed paths are publicly readable in robots.txt, so listing a private directory there advertises its location to anyone who looks. robots.txt is a crawl-management tool, not a security control and not a removal guarantee.

Declaring your sitemap

Add a Sitemap: https://yourdomain.com/sitemap.xml line to the file. It is the simplest way to make sure every crawler, not just the ones you have verified a property with, can find your sitemap. The directive is independent of the user-agent groups above it, so its position in the file does not matter. The Kalenux audit flags a robots.txt that does not declare one.

Make sure robots.txt isn't blocking what matters

Free to start. Check your robots.txt against every indexable page on your site.

Originally published at https://kalenux.com.tr/seo-guide/crawlability/robots-txt-seo-guide

Thin Content: Why Low-Value Pages Do Not Rank (and What to Do)

Emir Baycan — Mon, 31 Aug 2026 23:00:33 +0000

Thin content is a page that adds too little value to earn a place in search. It is rarely about word count alone, and that is where most audits go wrong. Here is what actually signals thin content, why a pile of it drags down the pages you care about, and how to decide between improving, merging, or removing.

Thin is not the same as short

A 200-word answer that fully resolves the query is not thin. A 2,000-word page that restates the same point five different ways is. Search engines are judging whether the page satisfies the intent behind the query, not whether it hit a word target.

This matters because the usual fix, padding a short page until it looks substantial, makes the problem worse. You end up with more text and the same amount of information.

The signals that actually stack up

No single one of these condemns a page. Several together is the pattern to look for:

Duplicated or near-duplicated body copy across many URLs, usually from templating
No original information: nothing on the page that is not already on the pages it links to
Auto-generated or spun text, including pages built purely from a database field
Doorway pages: near-identical pages targeting location or keyword variants
Affiliate pages that reproduce a manufacturer description with nothing added
High bounce with near-zero dwell time on a page that ranks but does not hold anyone

Why a few thin pages become a site problem

Crawl budget is the mechanism people cite most, and it is real on large sites: crawlers spend time on low-value URLs instead of the ones you want indexed.

The bigger effect is dilution. When many URLs on a domain compete weakly for related terms, none of them accumulates the signals that would make one of them rank well. You are splitting authority across pages that individually cannot earn it.

Improve, merge, or remove

Three options, and the right one depends on whether the page has anything worth keeping:

Improve when the topic deserves a page and the execution is the problem. Add the specifics that are genuinely missing: real numbers, a worked example, the edge case nobody documented.

Merge when several pages orbit one topic. Consolidate into the strongest URL and 301 the rest. This is usually the highest-return move, because it concentrates signals instead of spreading them.

Remove when the page serves no one. Return 410 if it should never come back, or 301 to the closest genuinely relevant page. Do not redirect everything to the homepage: that reads as a soft 404 and helps nobody.

How to find them

Sort your pages by organic entrances over the last 90 days and look at the bottom. Cross-reference against word count and internal links pointing in. Pages with no entrances, few inbound internal links, and body copy that appears on other URLs are your candidates.

Then read a sample by hand. Metrics identify suspects; only reading tells you whether the page has something worth saving.

Full version with the audit checklist: Thin Content: Why Low-Value Pages Do Not Rank

What Is a Canonical Tag (and the Mistakes That Deindex Pages)

Emir Baycan — Mon, 31 Aug 2026 15:00:54 +0000

A canonical tag tells search engines which URL is the master copy of a page. Used correctly it consolidates your ranking signals; used wrong it silently removes pages from search. Here's how it works and the mistakes that deindex pages.

What a canonical tag does

When the same content is reachable at more than one URL, the canonical tag names the one true version, so search engines consolidate signals onto it instead of splitting them.

A canonical tag is a single line of HTML you place in the of a page. It names the URL that should be treated as the master, preferred version of that content. It looks like this:

In the of the page

Why is it needed at all? Because the same page is almost always reachable at several different URLs without you realising. Each of these can return the identical content, and to a search engine they look like separate, duplicate pages competing with each other:

All of these can serve the same pagehttps://example.com/blue-widgetshttps://example.com/blue-widgets/← trailing slashhttp://example.com/blue-widgets ← HTTP not HTTPShttps://www.example.com/blue-widgets ← www varianthttps://example.com/blue-widgets?ref=twitter← tracking parameter

Left unmanaged, Google has to guess which of these is the "real" one, and your ranking signals (links, content quality, click data) get split across the duplicates instead of pooling on one strong page. The canonical tag removes the guesswork: every variant declares the same master URL, and the signals consolidate onto it. For the vast majority of indexable pages, the correct master is the page itself, a self-referencing canonical.

The self-referencing canonical (your default)

Almost every indexable page should point its canonical at its own URL. This is the single most important canonical habit.

A self-referencing canonical explicitly confirms "this URL is the preferred version," which protects the page from a parameter or duplicate URL being chosen instead. So on the page at https://example.com/blue-widgets, the canonical in its head should be its own clean URL:

On https://example.com/blue-widgets

Always use the absolute URL (full https:// and domain), not a relative path, and match your site's exact trailing-slash and www convention. A relative or inconsistent canonical is one of the easiest ways to send a mixed signal.

The canonical mistakes that deindex pages

Each of these is a real check in the audit, because each one quietly removes pages from search.

Canonical points to a redirect

If a page's canonical points to a URL that 301-redirects somewhere else, you're sending search engines on a detour. Google often ignores a canonical that resolves through a redirect and picks its own, which may not be the URL you wanted. Point the canonical at the final destination directly.

Canonical points to a 4XX or 5XX

A canonical that points to a 404 or a server error is naming a dead page as the master. Google can't honour it, so it disregards the signal and guesses. The page's ranking signals scatter. The audit flags canonicals pointing to 4XX and 5XX targets specifically.

Every page canonicalising to page 1

A frequent and costly bug: paginated archive pages (page 2, 3, 4...) all set their canonical to page 1. This tells Google that pages 2+ are duplicates of page 1, so it deindexes them: along with the links and products only reachable from those pages. Paginated pages should self-reference their canonical.

Canonical to the wrong protocol or domain

A canonical pointing from HTTPS to HTTP (or vice versa), or off to a different domain, sends authority and indexing intent to the wrong place. The audit flags canonical HTTP-to-HTTPS, HTTPS-to-HTTP, and off-domain canonicals. After any HTTPS or domain migration, re-check that every canonical references the new, correct URL.

Wrong: deindexes pages 2+

Every paginated page canonicalising to page 1:

Right: each page self-references

Paginated pages point to themselves:

What this looks like in Google Search Console

When a canonical points away from a page, Search Console's URL Inspection tool shows a mismatch: User-declared canonical is the URL you set, but Google-selected canonical differs, or the page lands in the "Alternate page with proper canonical tag" or "Duplicate, Google chose different canonical than user" report under Pages. If a page you want indexed shows either of those, its ranking signals are being sent elsewhere. That is the symptom this whole section is about.

Canonical best practice

A short set of rules that prevents almost every canonical problem.

Self-reference by default. Every indexable page should canonicalise to itself unless it's genuinely a duplicate of another URL. Point at a live, canonical 200 URL. Never canonicalise to a redirect, an error page, or a non-canonical URL. Keep it absolute and consistent. Use the full URL with the correct protocol, domain and trailing-slash convention. Align canonical, hreflang and internal links. They should all reference the same URL for a page, if your hreflang points one way and your canonical another, you've sent a contradiction. Re-check after migrations, when canonicals most often break.

Canonicals in practice

A strong hint, not a hard directive

A canonical tag is a strong hint rather than a binding instruction. Google usually honours a clear, consistent canonical, but where your signals conflict, the canonical says one thing while internal links and the sitemap say another, it is free to choose a different URL entirely.

What makes a canonical reliable, then, is consistency rather than the tag on its own. Canonical, internal links, sitemap and hreflang should all name the same URL for a page. When they agree, Google has no reason to override you; when they disagree, the canonical is just one vote among several.

Does every page need a canonical?

A self-referencing canonical on every indexable page is best practice, and the simplest policy to apply consistently. It is not strictly mandatory, but it removes ambiguity and protects the page against a parameter variant or duplicate URL being selected in its place.

The audit flags non-canonical URLs that receive organic traffic, which is usually the visible symptom of a missing or wrong canonical: a variant URL is ranking where the master should be, which means the signals are pooling on the wrong page.

Choosing between a canonical and noindex

Use a canonical when two URLs are duplicates and you want to consolidate their signals onto one. Use noindex when you want a page kept out of search entirely.

They solve genuinely different problems: canonical merges, noindex excludes. Do not combine a noindex with a canonical pointing to another URL, because the two directives contradict each other, one saying "this page is a copy of that one" and the other saying "remove this page", and Google is left to resolve a conflict you created.

Choosing between a canonical and a 301 redirect

If users never need to see the duplicate URL, an old path or a non-www version being the obvious cases, use a 301 redirect. It is the stronger signal and it sends both users and crawlers to the right place in one step.

Reach for a canonical when the duplicate URL must stay accessible to users but should not compete in search: a print version, or a page reachable through filters or tracking parameters. The rule of thumb is short: redirect if the duplicate should not exist, canonical if it must exist but should not rank separately.

Handling tracking parameters like ?ref= and ?utm_

Give the page a self-referencing canonical pointing at the clean, parameter-free URL. Every ?utm_source=... or ?ref=... variant then declares that same canonical, so the analytics parameters never spawn duplicate pages in the index.

This is the standard fix for parameter-driven duplication, and it is worth putting in place before you start any campaign that appends tracking parameters at scale, since the duplicates otherwise accumulate quietly in the background.

Catch broken canonicals before they deindex you

Free to start. Find canonicals pointing to redirects, errors and the wrong URLs across your site.

Originally published at https://kalenux.com.tr/seo-guide/indexability/what-is-a-canonical-tag

noindex Explained: When to Use It and When It Kills Rankings

Emir Baycan — Thu, 27 Aug 2026 14:40:56 +0000

noindex is the right way to keep a page out of search, and one of the most dangerous tags in SEO when it ends up where it shouldn't. Here's noindex explained: when to use it, when it silently kills rankings, and how to catch a stray one fast.

What noindex actually does

noindex tells search engines: crawl this page if you like, but don't put it in search results. It's a precise tool for excluding pages from the index without hiding them from users.

You can apply noindex two ways. The common one is a meta robots tag in the page ; the other is an HTTP response header, which works for non-HTML files like PDFs where you can't add a meta tag:

Method 1, meta tag in the Method 2, HTTP response header (e.g. for a PDF)X-Robots-Tag: noindex

Either way, search engines that crawl the page will drop it from their index. The critical catch: the page must be crawlable for noindex to work. If it's blocked in robots.txt, Google never fetches the page, never sees the tag, and the URL can stay indexed. Use robots.txt for crawling and noindex for indexing; they are not interchangeable, and combining them backfires.

When to use noindex, and when not to

The line between a smart noindex and a costly one.

Good uses of noindex

Thank-you and order-confirmation pages, internal search-results pages, thin tag/filter/pagination pages that add no unique value, staging or duplicate utility pages, and private account or admin areas. These are pages users may need but that would clutter or dilute your presence in search. noindex keeps them accessible while out of the index.

Don't use noindex for duplicates you want merged

If two URLs are duplicates and you want their ranking signals combined onto one, use a canonical tag, not noindex. noindex removes a page entirely; canonical consolidates it. Mixing them, a noindex plus a canonical to another URL: sends conflicting signals.

Never let noindex reach pages you want ranked

This sounds obvious, but it's the single most common SEO disaster, because noindex usually arrives by accident via a template or global setting, not a deliberate page-level choice. The next section covers exactly how that happens.

The stray-noindex disaster (and how to catch it)

How a single tag deindexes a whole site, and the check that stops it.

Most sites are built with shared templates. If a noindex tag lands on a template, a staging-environment "noindex everything" rule that ships to production, a CMS visibility toggle left on, a plugin default, or a developer's leftover, it doesn't deindex one page. It deindexes every page using that template at once, a whole blog, a whole product category, sometimes the entire site. The cruel part is the delay. Nothing breaks visibly; the pages load fine. Then over the following days, as Google re-crawls and honours the tag, rankings and traffic collapse, and by the time someone notices the traffic drop, the cause is buried days in the past.

The defence is monitoring for change. An audit that flags pages which became non-indexable since the last crawl turns a silent, delayed catastrophe into an alert the same day the tag ships. Run a crawl after every deploy that touches templates, and the stray noindex never gets the chance to do real damage.

noindex in practice

noindex compared with a robots.txt disallow

These two look interchangeable and are not. noindex keeps a page out of the index while still allowing crawling, which is exactly what you want when the goal is removal from search. A robots.txt Disallow does the opposite: it blocks crawling but does not remove a page from the index, and worse, it stops Google ever seeing a noindex tag you add later.

The practical rule is to allow crawling and add noindex when you want a page out of search, and to reserve robots.txt for managing crawl load. Applying both to the same URL is the combination that leaves pages stubbornly indexed with no description attached.

The difference between noindex,follow and noindex,nofollow

noindex,follow keeps the page out of search but still lets link equity flow through its outbound links. That is useful for pagination and utility pages you want excluded from results while still passing authority onward to the pages they link to.

noindex,nofollow excludes the page and also stops search engines following its links, cutting off that flow entirely. The audit distinguishes between the two variants precisely because choosing the wrong one can silently sever internal link flow through a whole section of a site.

How long does noindex take to take effect?

A noindex directive applies the next time Google crawls the page and processes the tag, which can be anywhere from hours to weeks depending on how often that page is crawled. To speed up removal of an important page, request indexing or use the removal tool in Search Console.

That same delay is what makes an accidental noindex so dangerous. The mistake ships on one day and the damage lands several days later, by which time the deploy that caused it is well behind you and no longer the obvious suspect.

Catch a stray noindex the day it ships

Free to start. Get alerted when any page becomes non-indexable since your last crawl.

Originally published at https://kalenux.com.tr/seo-guide/indexability/noindex-explained

We stopped paying $100+/mo for SEO tools. Here's the technical-audit setup we use instead

Emir Baycan — Sun, 07 Jun 2026 21:12:19 +0000

We run 10+ web products. For years our SEO tooling was a $100+/month subscription we mostly used for one job: technical audits. The keyword and backlink dashboards sat untouched while the bill renewed every month. At some point the math stopped making sense, so we rebuilt our stack around what we actually use. Here's the honest breakdown — what we kept free, what we replaced, and why.

What a technical audit actually needs to check

Before picking tools, it helps to know what you're auditing. The technical layer is finite and well-defined:

Crawlability — robots.txt, broken internal links, redirect chains
Indexability — noindex tags, canonical correctness, soft 404s, orphan pages
Sitemaps — only canonical, indexable, 200-status URLs
On-page — titles, meta descriptions, one H1, valid structured data
International — hreflang reciprocity (if you're multilingual)
Performance — Core Web Vitals (LCP, CLS, INP)

That's it. None of it requires a keyword database or a backlink index — the two things you're really paying $100+/mo for in the big suites.

The free tools that cover most of it

Google Search Console is non-negotiable and free. The Pages report is the source of truth for what's indexed and why the rest isn't. PageSpeed Insights (free) gives you real Core Web Vitals. Screaming Frog's free tier crawls up to 500 URLs, which is plenty for small sites.

For a long time that was our whole stack: GSC + PageSpeed + free Screaming Frog. If your site is under 500 pages, you may not need anything else.

Where it broke down for us

The free Screaming Frog cap (500 URLs) is the wall. Several of our sites are bigger, and re-auditing them meant either the paid Screaming Frog licence (annual) or going back to a monthly suite. Both felt wrong for something we run after every deploy.

So we built our own crawler for the technical layer — and then, honestly, turned it into a product because other people we talked to had the same problem. It does the whole checklist above across the entire site, plus PageSpeed, with one-time pricing instead of a subscription ($9-49, credits never expire). It's the same engine we run on our own properties — kalenux.com.tr/seo, and there's a free no-signup check if you want to see what it flags. It is not a keyword tool — we still use a separate (cheaper) option for that.

The actual takeaway

The point isn't our tool — it's the unbundling. Most people pay for a full SEO suite to use 20% of it. Split the jobs:

Technical audits → a dedicated crawler (free tier, a one-time tool, or paid Screaming Frog)
Keyword research → a cheaper standalone option, or free ones
Backlinks → only if you genuinely work on link-building

Most small sites and freelancers can cover everything they actually use for a fraction of a single suite subscription. We went from $100+/mo to a few dollars a year on the technical side, with no loss in what we could check.

Written by the Kalenux team.

I built 9 free SEO tools so I'd stop paying $100/mo to audit my own sites

Emir Baycan — Sun, 07 Jun 2026 20:05:54 +0000

We run 10+ web products. For years we paid for an SEO suite mostly to do one thing: technical audits. The keyword and backlink features sat unused while the bill renewed every month. So we built our own technical-audit engine — and then split the individual checks out into free, no-signup tools. Here's what each one does and the technical reasoning behind it.

All of them run an instant check on a URL with no account. Try them on your own site as you read.

The crawlability & indexing tools

Indexability Checker — the most important one. A page that isn't indexable can't rank, full stop. It checks for noindex tags, robots.txt blocks, canonical conflicts and non-200 status. The classic outage we've all caused: a stray noindex on a shared layout that ships to production and quietly deindexes a whole section.

Canonical Checker — verifies rel=canonical is self-referencing and resolves to a live 200. The subtle bug it catches: a canonical pointing at a URL that then redirects. Google ignores those and picks its own canonical, so your signal is wasted. Common after an HTTP→HTTPS migration where canonicals weren't updated.

Sitemap Validator — locates your sitemap (via robots.txt or /sitemap.xml), validates the XML, counts URLs, checks HTTPS consistency, and samples a few URLs for live 200 status. A sitemap full of redirected or 404 URLs sends mixed signals and wastes crawl budget.

Technical SEO Checker — the full crawl-based audit across every page, not just the homepage.

The on-page tools

Meta Tag Checker — title, meta description, canonical, viewport and Open Graph tags, checked for presence and length. Title truncates around 60 chars; meta description around 160.

Broken Link Checker — finds broken internal and outbound links plus redirect chains, with the source pages for each. Fixing broken links is the highest-return, lowest-effort technical task there is.

Hreflang Checker — for multilingual sites. Hreflang is the most error-prone area of technical SEO: it needs reciprocal return tags, a self-reference, valid language codes, and targets that return 200. Miss any and Google ignores the whole cluster.

Performance

PageSpeed Test — real Core Web Vitals (LCP, CLS, INP) on your key templates, mobile and desktop. Test a product or category page, not just the homepage — those carry the traffic and usually score worst.

Why free, why no signup

The instant checks read one page (your homepage) and need no account, because the fastest way to show a tool is useful is to just let people use it. The full site-wide crawl is the paid product, but it's one-time pricing ($9-49, credits never expire) rather than a subscription — which was the whole point of building it. We were tired of paying monthly to re-audit our own properties.

If you maintain your own sites, the full toolbox lives at kalenux.com.tr/seo-tools. And if you just want the methodology, the complete technical SEO audit guide walks through the whole process in order.

Written by the Kalenux team.

8 ways your site quietly deindexes itself (and how to catch them)

Emir Baycan — Fri, 05 Jun 2026 17:46:01 +0000

Deindexing is the quietest way to lose traffic. Nothing 404s, nothing looks broken - pages just slip out of Google's index and the rankings go with them. We run 10+ web products and these are the eight causes we see over and over. Each one is invisible until you go looking.

1. A stray `noindex` on a shared template

The classic. Someone adds <meta name="robots" content="noindex"> to a layout or component to hide a staging page, and it ships to a template used by hundreds of pages. Whole sections vanish from search.

Catch it: grep your built output for noindex. It should only appear where you genuinely want it (thank-you pages, internal search results). Anything else is a leak.

2. Canonical tags pointing to the wrong URL

If page A says <link rel="canonical" href="B">, you're telling Google "index B instead of me." Get this wrong across a template - e.g. every paginated page canonicalising to page 1 - and you deindex pages 2+.

Catch it: every page should self-reference its canonical unless you have a deliberate reason not to. Watch especially for canonicals pointing to a redirect or a 404 - Google ignores those and picks its own, unpredictably.

3. robots.txt blocking what you want indexed

A Disallow: rule meant for one folder can match more than you think. And blocking a URL in robots.txt doesn't deindex it - it just stops Google re-crawling, so a page can stay indexed with no snippet, or get stuck.

Catch it: test your important paths against your robots.txt rules. Make sure your sitemap is declared with a Sitemap: line too.

4. Non-canonical or redirected URLs in your sitemap

Your sitemap is a list of "please index these." If it's full of redirected URLs, non-canonical variants, or 404s, you send mixed signals and waste crawl budget.

Catch it: your sitemap should contain only canonical, indexable, 200-status URLs. Audit it.

5. Soft 404s

A page returns HTTP 200 but has no real content ("No results found", an empty category). Google calls these soft 404s and quietly drops them - or worse, loses trust in that section.

Catch it: thin/empty pages that return 200 are the tell. Either add content, noindex them deliberately, or return a real 404.

6. Redirect chains and loops

A -> B -> C -> D wastes crawl budget and dilutes signals at every hop. A redirect loop (A -> B -> A) is worse - Google gives up.

Catch it: collapse every chain to a single hop. A -> D, done.

7. Canonical-to-redirect

A subtle one: page A canonicalises to B, but B redirects to C. Now your canonical signal is broken and Google has to guess. This commonly happens after an HTTP-to-HTTPS or trailing-slash migration where canonicals weren't updated.

Catch it: make sure every canonical target returns 200 directly, not a redirect.

8. Orphan pages

A page with zero internal links pointing to it. Google can technically find it via the sitemap, but with no internal links it reads as low-priority and often won't get or keep an index slot.

Catch it: build your internal link graph and find pages with zero incoming links. One warning from experience: if your crawler times out while resolving the link graph, it will report every page as an orphan. Make sure the incoming-link count actually computed before you trust it - a false "everything is orphaned" result has sent people chasing ghosts.

How to monitor this without a $100/mo suite

Google Search Console -> Pages report is the source of truth: it tells you exactly what's indexed and, for everything that isn't, why. Free. Check it weekly.
Screaming Frog free tier crawls 500 URLs and surfaces most of the above for small sites.
For site-wide crawls without the subscription, we built our own (Kalenux SEO Audit) - it flags noindex, canonical errors, soft 404s, redirect chains, sitemap mismatches and orphans across the whole site, one-time pricing instead of a monthly bill. It's the technical-audit piece only, not a keyword tool.

Deindexing is sneaky precisely because nothing looks broken. Set up the monitoring, run a crawl after every deploy that touches templates or redirects, and you'll catch these before they cost you a quarter of rankings.