Security in the AI Era: Every Request Has a Cost — Three Layers of Defense for Your Application

Security in the AI Era: Every Request Has a Cost

Relnx is a small product. We are not a bank, we don't run a crypto exchange, and nobody has a reason to target us specifically. And yet, in a single ordinary week, about one in eight requests that reached our edge was a scanner, a bot or an attack probe — roughly 112,000 requests, of which more than 9,000 were blocked by firewall rules, over 1,200 were rate-limited and more than 3,000 were sent a bot challenge.

None of those requests found a leak. That is the point of this post: in the AI era, the question is no longer "does my application have a vulnerability?" The question is how much the constant background noise costs you — in compute, in money, in alert fatigue, and sometimes in a real customer who couldn't load your page because your server was busy answering a bot.

This post covers what we actually saw in our own traffic, why we think it is growing so fast, the three layers of defense we recommend (edge, load balancer, and the Kubernetes runtime with Tetragon), and how an AI agent fed with the right signals can keep those defenses sharp without a human staring at dashboards all day.

It's not about whether you have a leak

The traditional mental model of web security is binary: either the attacker finds a hole or they don't. If your .env file isn't publicly served, the request for /.env fails and no harm is done.

That model misses the cost of the request itself. Every hostile request that reaches your origin:

• Consumes compute. A single-page app often answers any unknown path with 200 OK and the full HTML shell. A scanner asking for /.git/config gets a perfectly valid page back — it learned nothing, but your server rendered and shipped it anyway. Multiply that by thousands of paths per scanner.
• Competes with real users. One cloud IP requested the same blog page 114 times in a single minute on our site. Pods have finite CPU, connection pools have finite slots, and databases have finite connections. When a burst lands at the wrong moment — during a deploy, or while a node is being replaced — the real visitor in the same minute sees a timeout or a 504.
• Triggers your own defenses against your customers. Rate limits are shared resources too. We once saw a real visitor rate-limited 22 times because they opened several pages at once and our frontend was chatty. The limit was there because of bots; the victim was a customer.
• Pollutes your observability. Every probe is a log line, a trace, a metric data point. It costs money to ingest, and worse, it buries the one error that actually matters under thousands that don't.
• Is reconnaissance. A request that fails today tells the scanner what stack you run. The 404 pattern, the headers, and the framework fingerprints all feed the next, more targeted attempt.

Security in this era is as much about capacity and signal quality as it is about vulnerabilities.

What we actually saw

We run a security and performance review of our edge, load balancer and backend every few hours. Here are the patterns that kept showing up.

1. Secret hunting at industrial scale

The single biggest category is automated searching for leaked configuration and credentials. Typical paths from a single scanner session:

/.env  /.env.save  /.env.production  /api/.env  /backend/.env  /docker/.env
/.git/config  /.git/HEAD  /.git/COMMIT_EDITMSG
/secrets.yml  /values.yaml  /serverless.yaml  /docker-compose.override.yml
/firebase-service-account.json  /firebaseConfig.js  /sa.json  /private-key
/.npmrc  /.bash_history  /sftp-config.json  /backup.sql
/.claude/settings.json
/static/..;/..;/..;/proc/self/environ

Two things stand out. First, the list keeps evolving with the ecosystem: AI tooling config files like .claude/settings.json are now standard entries, right next to cloud service-account keys. Second, the path-traversal tricks (..;/) are specifically designed for proxies and servlet containers that normalize paths differently from the app behind them.

2. Fake AI crawlers

Everybody wants to appear in AI search answers, so many sites deliberately let AI crawlers through. Attackers noticed. We saw a sustained campaign of requests from cloud provider IPs whose User-Agent claimed to be GPTBot, ClaudeBot, PerplexityBot, Googlebot, Amazonbot, DuckAssistBot, Meta's fetcher and a dozen other AI and search crawlers — none of which were the real crawler. Within a single session, one IP rotated through KimiBot, Qwenbot, Hunyuan, Bravebot and MoonshotBot user agents while asking for .env files.

A User-Agent string is free text. Trusting it is the same as trusting a form field.

3. Host hopping

When we blocked one scanner on our main website, it moved to our API hostname and then to every other subdomain it could find in DNS and certificate transparency logs, replaying the same payloads. A defense that only covers www is a defense that covers one door.

4. Volume on cheap pages

Not every bot is clever. Some just hammer one public page — the blog index, the homepage — at a rate no human can produce. These rarely look malicious per request, which is exactly why they slip past path-based rules.

5. Not everything automated is hostile

A payment provider's merchant security scanner checked our site for exposed files. An attack-surface-management vendor scanned us from its published ranges. Real search and AI crawlers indexed thousands of pages a day. Blanket-blocking "bots" is not the answer — you would lose your search presence and break your integrations. The job is to tell them apart.

Why this is increasing so fast

We don't have a crystal ball, but the drivers are visible in the traffic itself:

• The cost of writing and running a scanner has collapsed. Generating a list of 500 framework-specific sensitive paths, writing the HTTP client that rotates user agents, and adapting to each response used to take skill and time. Now it is a prompt. Agent-style tools can fingerprint a stack (we saw requests for Vite and React Server Components internals like __vite_rsc_findSourceMapURL and __rsc) and adjust the next batch of probes accordingly.
• Secrets became directly monetizable. A leaked LLM API key is spendable money the moment it's found — attackers resell access or run their own workloads on your bill. The value of a single .env file went up, so the volume of searching for them went up.
• AI crawlers created a trusted disguise. Sites relaxed rules to let AI crawlers in. Spoofing those crawlers is the cheapest way to inherit that trust.
• Cloud IPs are cheap and disposable. Most hostile traffic we saw came from major cloud providers, from addresses that appear for a few hours and never return. IP blocklists alone age out in days.
• New attack surface appears with every new tool. AI assistant configs, MCP server configs, serverless manifests, Helm values files — each new tool in the developer workflow adds a new file someone might accidentally deploy, and scanners add it to their list within weeks.

The result is a permanent, rising background radiation of automated traffic that every public application absorbs, regardless of size.

Three layers of defense

The principle is simple: reject bad traffic as early and as cheaply as possible, and assume each layer will occasionally miss. A request blocked at the edge costs you nothing. A request blocked at the load balancer costs a little. A request that reaches your pod costs the most — and if it ever succeeds, the runtime layer is your last line.

Layer 1: The edge (CDN / WAF)

This is where most of the volume should die. The examples below use Cloudflare's rule language because that is what we run, but every major CDN/WAF has equivalents.

Block known-bad paths before they reach you. Credential files, VCS metadata, CMS endpoints you don't run, and traversal patterns:

(http.request.uri.path contains "/.env")
or (http.request.uri.path contains "/.git/")
or (http.request.uri.path contains "wp-login.php")
or (http.request.uri.path contains "xmlrpc.php")
or (http.request.uri.path contains "/proc/self/")
or (http.request.uri.path contains "..;/")
or (ends_with(http.request.uri.path, "service-account.json"))

Always test new substrings against your own real URLs first. We once blocked a legitimate product page because its slug contained a word that also appears in credential filenames.

Verify crawlers instead of trusting their user agent. Block any request that claims to be a known search or AI crawler but isn't verified by your edge provider:

not cf.client.bot
and (
  lower(http.user_agent) contains "gptbot"
  or lower(http.user_agent) contains "claudebot"
  or lower(http.user_agent) contains "googlebot"
  or lower(http.user_agent) contains "perplexitybot"
  # ...the rest of the crawlers you care about
)

Some crawlers aren't in your provider's verified list but publish their IP ranges (Perplexity does, for example). Exempt them by published IP plus user agent, never by user agent alone — and re-check the published list periodically.

Rate-limit, but design the limit around real users. Per-IP request limits catch the hammering bots. Two lessons we learned the hard way:

• Don't count CORS preflight (OPTIONS) requests against the limit. A browser sends one before many API calls, so a legitimate page can double its request count without the user doing anything.
• If real users hit the limit, the fix is usually in your app, not the limit. Our feature page made three separate API calls per item. We replaced them with a single batched endpoint, and the false positives disappeared without loosening the limit.

Challenge suspicious clients, but watch for loops. Bot challenges are great for headless scanners with no user agent. They can also trap real users behind corporate proxies in a challenge loop. Count those loops; if they recur, your bot mode is too aggressive for your audience.

Put admin surfaces behind identity, not just obscurity. An IP allowlist is a good start. An identity-aware proxy (Cloudflare Access, Google IAP, an OIDC-protected gateway) is better, because people change networks. Remember that if your admin frontend calls your public API host, the admin API paths on that host need the same protection.

Cache public HTML at the edge. The cheapest request to serve is the one your origin never sees. Even a few hours of edge caching for marketing and blog pages absorbs most of the "hammer one page" bots.

Layer 2: The load balancer / ingress

The edge will miss things, and attackers will try to bypass it entirely by hitting your origin IP directly. The load balancer is where you enforce that everything came through the front door, and where you add a second, cheaper filter.

Lock the origin to your edge. Only accept traffic from your CDN's IP ranges (security groups / firewall rules), and ideally require mutual TLS between edge and origin (Cloudflare calls this Authenticated Origin Pulls). If someone finds your origin IP, they should get nothing.

Add a second line of limits and timeouts. An example with NGINX:

# Per-client limits (use the real client IP header from your CDN)
limit_req_zone  $binary_remote_addr zone=perip:10m rate=20r/s;
limit_conn_zone $binary_remote_addr zone=conn:10m;

server {
    # Slow-client protection
    client_body_timeout   10s;
    client_header_timeout 10s;
    client_max_body_size  1m;

    limit_req  zone=perip burst=40 nodelay;
    limit_conn conn 20;

    # Never serve dotfiles or VCS metadata, even if the edge misses them.
    # Answer with a cheap 404 instead of letting the SPA fallback render a page.
    location ~ /\.(?!well-known) { return 404; }
    location ~* \.(env|ya?ml|sql|bak|old|log)$ { return 404; }
}

The most important line here is the cheap 404. A single-page app that returns the full HTML shell for every unknown path is doing real work for every scanner request. Answering sensitive patterns directly at the load balancer is both safer and cheaper.

If you run Kubernetes, note that the community ingress-nginx controller has been retired. The same ideas apply to Gateway API implementations like Envoy Gateway, or to whatever L7 load balancer you use: rate limits, connection limits, timeouts, path denials, and origin locking.

Layer 3: The cluster runtime with Tetragon

The first two layers reduce volume. The third assumes that one day something will get through — a vulnerable dependency, a deserialization bug, a leaked token — and asks: if an attacker gets code execution inside a pod, what can they actually do?

Tetragon is a CNCF project from the Cilium team that uses eBPF to observe and enforce security policy inside the Linux kernel: process execution, file access, network connections and privilege changes, tagged with the Kubernetes pod and workload they came from. Because enforcement happens in the kernel, it can stop an action synchronously instead of alerting after the fact.

A practical starting policy: nothing in your backend pod except the application binary itself should read the Kubernetes service-account token or sensitive system files. If anything else tries, kill it:

apiVersion: cilium.io/v1alpha1
kind: TracingPolicyNamespaced
metadata:
  name: protect-sensitive-files
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: backend
  kprobes:
    - call: "security_file_permission"
      syscall: false
      args:
        - index: 0
          type: "file"
        - index: 1
          type: "int"
      selectors:
        - matchArgs:
            - index: 0
              operator: "Prefix"
              values:
                - "/var/run/secrets/kubernetes.io/serviceaccount"
                - "/etc/shadow"
          matchBinaries:
            - operator: "NotIn"
              values:
                - "/app/server"
          matchActions:
            - action: Sigkill

Start in observe-only mode: drop the Sigkill action and watch the events for a week with tetra getevents -o compact to see what legitimately touches these paths. Then enforce. Process execution is also worth watching from day one. A shell spawning inside a distroless Go container is almost never legitimate.

Tetragon works best alongside the basics it cannot replace:

• Least-privilege pods: non-root, read-only root filesystem, no privilege escalation, dropped capabilities, distroless images with no shell.
• Egress lockdown: a NetworkPolicy (or CiliumNetworkPolicy) that only lets the backend talk to its database and the few APIs it needs. Most real-world breaches end with data or credentials leaving the cluster, so make "leaving" hard.
• No default service-account token where the workload doesn't need the Kubernetes API (automountServiceAccountToken: false).
• More than one replica with a PodDisruptionBudget, so bot bursts during a node replacement or deploy degrade capacity instead of causing an outage.

Observability and an AI agent that keeps tuning

Three layers are not a one-time setup. Attackers rotate IPs, invent new paths and copy new user agents every week, and your own app changes too. Rules that were right last month produce false positives this month. This is where most teams fall behind: nobody has time to read WAF logs every day.

It is also where AI agents are genuinely useful — if, and only if, they have the right signals.

The signals that matter

• Edge security events: which rule fired, the action (block, challenge, rate-limit, skip), path, host, user agent, client network, and whether the edge verified the client as a known bot.
• Load balancer logs: status codes, upstream latency, and requests that reached the origin without coming through the edge.
• Application RED metrics: request rate, error rate and latency per route, so you can see when bot traffic turns into real-user pain.
• Runtime events: Tetragon process, file and network events from your workloads.
• Customer-journey checks: signup, login and payment success rates and latency, plus synthetic checks of key pages. These turn "lots of blocks" into "did any real customer notice?"

How we run it

We run an AI agent on a schedule — every few hours — with read access to these signals. Each run it compares the window with the baseline, investigates anything unusual, and reports in plain language: what was blocked, whether any real customers were affected, and what needs a human.

Some real things it caught for us:

• The fake-crawler campaign described above. It noticed "Googlebot" requests from cloud IPs that the edge hadn't verified, correlated them with .env probes, and proposed the verified-crawler rule.
• A false positive from that same rule. A legitimate AI crawler that isn't on our provider's verified list started getting blocked. The agent found the crawler's published IP list and proposed a narrow exemption — IP range and user agent — instead of removing the rule.
• A real customer being rate-limited. Instead of raising the limit, it traced the cause to our frontend making several API calls per item. The fix was a batched endpoint in the app, and the limit stayed where it was.
• A scanner that got 200 OK responses. It checked the response bodies before raising an alarm: they were the SPA's HTML shell, not leaked files. Then it blocked the scanner as it moved from host to host.

Guardrails make it safe

An agent that can change your firewall is itself a security risk, so we drew clear lines:

• Tightening is pre-approved, loosening is not. Blocking a new hostile IP or adding a narrow false-positive fix is fine. Disabling a rule, widening an allowlist or touching admin protection always needs a human's explicit yes.
• Back up, test, log. Before every change the agent saves the current ruleset. After the change it verifies that real pages still load, and it records what changed and why in a runbook that lives in the repository.
• Least-privilege credentials. The agent's API token can edit firewall rules and read analytics — nothing else, and only from known networks.
• Treat traffic as data, never as instructions. User agents, URL paths and request bodies are attacker-controlled text. An agent reading logs must never act on instructions it finds inside them, since prompt injection through a User-Agent header is entirely possible.

The result is not an autonomous security robot. It is a tireless analyst that reads every signal, every few hours, and brings a human only the decisions that need one.

A checklist to start this week

1. Measure first. What share of your traffic is verified bots, unverified automation and humans? You can't tune what you haven't counted.
2. At the edge: block credential/VCS/CMS paths, block unverified "crawlers", add a per-IP rate limit that ignores OPTIONS, cache public HTML, and put admin behind identity.
3. At the load balancer: lock the origin to your edge, add a second set of limits and timeouts, and return a cheap 404 for sensitive patterns instead of your SPA shell.
4. In the cluster: run Tetragon in observe mode, then enforce file and exec policies; lock down egress; run at least two replicas with a PDB.
5. Close the loop: wire edge, LB, app and runtime signals into one place, and let an agent review them on a schedule — with tight guardrails on what it may change.

Conclusion

In the AI era, being attacked is not an event; it is the weather. Scanners, fake crawlers and agent-driven reconnaissance will hit your application whether or not you have anything worth stealing, and every one of those requests costs you something — capacity, money, signal quality, and occasionally a real customer.

The answer isn't paranoia. It's layers: drop the noise at the edge, enforce the basics at the load balancer, contain the worst case in the kernel with Tetragon, and keep the whole system tuned with good observability and an AI agent that never gets tired of reading logs.

The tools in this stack — Cilium, Tetragon, Envoy Gateway, your ingress controller, your WAF provider — ship security fixes constantly, and an out-of-date security layer is a weak one. Relnx tracks releases and security updates for the cloud-native tools you run and tells you when something important changes, so your defenses stay current. Start tracking your stack for free.