Most teams treat security headers as a compliance checkbox: run a scanner, get a letter grade, add whatever headers get you from a C to an A, move on. That framing misses something worth knowing, which is that several of these headers also shape how crawlers and browsers evaluate your site in ways that show up outside any security audit.
Content-Security-Policy can silently break rendering
A Content-Security-Policy header that's too restrictive can block inline scripts or styles your page actually needs, which is a problem you'll usually notice fast in a browser but can be much harder to catch when it's a crawler rendering the page instead of a human. Google renders pages the way a browser would before judging their content quality, so a CSP that blocks a script your layout depends on can produce the same "page looks broken" outcome for a crawler that a misconfigured robots.txt rule can, just through a completely different mechanism.
Referrer-Policy affects your own analytics
Referrer-Policy controls how much of the referring URL gets passed along when someone clicks a link from your site to another one, or from another site to yours. Set it too aggressively (no-referrer everywhere) and you lose visibility into which of your own pages are driving outbound clicks in your analytics. Set it too permissively and you might be leaking full query-string URLs, including anything sensitive that ended up in a URL parameter, to third-party sites your users click through to. strict-origin-when-cross-origin is a reasonable default for most sites: full referrer on same-origin navigation, origin-only across origins.
X-Frame-Options and embeddability
If you ever want your own content embeddable in an iframe elsewhere (a widget, a partner integration, a preview card), a blanket X-Frame-Options: DENY will block that entirely, with no partial-allow option. Content-Security-Policy's frame-ancestors directive is the more flexible modern replacement, letting you allowlist specific origins instead of an all-or-nothing rule. Worth checking which one your stack is actually sending, since a lot of default configs still ship the older, less flexible header.
HSTS has a real, sticky downside if you get it wrong
Strict-Transport-Security tells browsers to only ever connect to your domain over HTTPS, for a duration you specify, and browsers cache that instruction aggressively. If you set a long max-age and then later need to run a subdomain over plain HTTP for some reason, users who already cached the HSTS policy simply can't reach it until the policy expires, no matter what your server does. Set it deliberately, understand the includeSubDomains and preload flags before turning them on, and don't copy a max-age value from a template without knowing what you're committing to.
Permissions-Policy can quietly disable embeds you rely on
Permissions-Policy (the successor to Feature-Policy) lets you turn off browser features like camera, microphone, geolocation, or autoplay for your own page and for anything embedded in it. Teams usually add it for the obvious reason: lock down features you don't use so a compromised third-party script can't quietly turn on the microphone. That part works as advertised and is worth doing.
Where it gets messy is with third-party embeds you actually want. A YouTube video embed expects autoplay and sometimes fullscreen to be allowed for the frame it sits in. A payment widget or an identity verification vendor often needs camera or publickey-credentials-get passed through explicitly, because permissions don't propagate to iframes by default under a restrictive policy. If you write a blanket Permissions-Policy that disables everything and never add the specific allow= exceptions those embeds need, the embed doesn't error loudly, it just silently fails to do the one thing it was there for, like a video that never autoplays or a scanner that never gets camera access. Debugging that is annoying precisely because the page looks fine, nothing throws a console error a casual glance would catch, and the feature just doesn't work.
The fix is boring but effective: whenever you add a third-party embed, check its documentation for which permissions it expects the parent frame to delegate, and add those explicitly rather than copying a maximally locked-down policy from a template and assuming it'll be fine.
Cross-Origin-Resource-Policy and Cross-Origin-Opener-Policy can break resource loading
Cross-Origin-Resource-Policy (CORP) and Cross-Origin-Opener-Policy (COOP) exist mainly to isolate your page from cross-origin attacks like Spectre-style side channels, and browsers have been pushing sites toward setting them more aggressively over the last few years. The side effect that catches people off guard is that both headers can block perfectly legitimate cross-origin loads if the values don't match what the other side expects.
Set Cross-Origin-Resource-Policy: same-origin on an asset your own CDN serves from a different subdomain than your main site, and the browser will refuse to let your main page load it, because from the browser's point of view that's a different origin making the request. The asset returns fine over the network, dev tools show a 200, and the resource still never renders because the policy blocked it after the fact. The usual fix is cross-origin or same-site instead of same-origin, depending on how your subdomains are actually laid out.
COOP causes a subtler version of the same problem with popups and OAuth flows. A strict Cross-Origin-Opener-Policy: same-origin can sever the window.opener reference a login popup relies on to communicate back to the page that opened it, which breaks the "log in via popup, then refresh the parent page" pattern some auth providers use. If you support any third-party login flow that depends on window messaging between a popup and its opener, test that flow specifically after tightening COOP, because it's not the kind of break that shows up in a normal page load.
CORS headers and CDN caching don't always agree
Access-Control-Allow-Origin and CDN caching interact in a way that trips people up because each system is reasoning about the request independently. If your CDN caches a response and that cached response included an Access-Control-Allow-Origin header scoped to one specific origin, every subsequent request for that same cached asset gets served with the same CORS header, regardless of which origin is actually asking for it this time.
That means a font or JSON file that legitimately needs to be fetchable from multiple origins can end up cached with only the first requesting origin's value baked into the response, silently breaking cross-origin fetches from every other origin until the cache expires or gets purged. The standard fix is to add Vary: Origin alongside your CORS header so the CDN treats different origins as different cache entries instead of collapsing them into one. It costs you some cache efficiency in exchange for correctness, which is usually the right trade for anything served to more than one origin.
Where this connects back to crawling
Headers and crawl directives are two separate mechanisms that occasionally interact in confusing ways, similar to how robots.txt and a page's noindex meta tag are separate mechanisms people conflate. I wrote a longer breakdown of the robots.txt side of that confusion here, including why blocking a folder in robots.txt doesn't remove already-indexed pages the way people expect.
I run new deployments through EvvyTools' HTTP Security Header Grader alongside a normal Lighthouse pass, mostly because it catches the headers that don't show up in a standard performance audit at all.
For deeper reference, MDN's HTTP headers documentation covers every header mentioned here in more depth, OWASP's Secure Headers Project has a solid checklist if you're setting these up from scratch, and the W3C's CSP specification is the authoritative source if you need to resolve an edge case the summaries don't cover.
Top comments (0)