<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Nachiket Gholap</title>
    <description>The latest articles on DEV Community by Nachiket Gholap (@nachiketgholap).</description>
    <link>https://dev.to/nachiketgholap</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4064703%2F7a5487a3-78ce-4dab-8bc5-682e366fb68a.png</url>
      <title>DEV Community: Nachiket Gholap</title>
      <link>https://dev.to/nachiketgholap</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/nachiketgholap"/>
    <language>en</language>
    <item>
      <title>I Blocked the Attacker's IP. The Attack Didn't Stop. Here's Why.</title>
      <dc:creator>Nachiket Gholap</dc:creator>
      <pubDate>Wed, 18 Nov 2026 10:18:55 +0000</pubDate>
      <link>https://dev.to/nachiketgholap/i-blocked-the-attackers-ip-the-attack-didnt-stop-heres-why-dp4</link>
      <guid>https://dev.to/nachiketgholap/i-blocked-the-attackers-ip-the-attack-didnt-stop-heres-why-dp4</guid>
      <description>&lt;p&gt;It's 2:14 AM and your phone won't stop buzzing.&lt;/p&gt;

&lt;p&gt;The alert says your &lt;code&gt;/reset-password&lt;/code&gt; endpoint is getting &lt;strong&gt;40,000 requests per second&lt;/strong&gt;. Normal traffic is maybe a few requests per minute. Your dashboard shows it all coming from one IP address, somewhere in India.&lt;/p&gt;

&lt;p&gt;Easy fix, right? You add that IP to the block list, refresh the dashboard, and wait for the graph to drop.&lt;/p&gt;

&lt;p&gt;It doesn't drop.&lt;/p&gt;

&lt;p&gt;It's still 40,000 requests per second.&lt;/p&gt;

&lt;p&gt;I came across this as a system design interview question, and it stuck with me, because the "obvious" answer is the wrong one. I'm a Business Analyst who's learning system design, not a security engineer. So in this post I'm working through it the way I'd want someone to explain it to me: what's actually going on, why the block didn't work, and what a calmer, layered response looks like.&lt;/p&gt;

&lt;p&gt;Let's take it apart.&lt;/p&gt;

&lt;h2&gt;
  
  
  Understand the Problem First
&lt;/h2&gt;

&lt;p&gt;Before touching any config, it helps to slow down and ask: what is this endpoint, and why would anyone hammer it?&lt;/p&gt;

&lt;h3&gt;
  
  
  What &lt;code&gt;/reset-password&lt;/code&gt; actually does
&lt;/h3&gt;

&lt;p&gt;When you click "Forgot password?", the app usually:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Takes the email or phone number you typed&lt;/li&gt;
&lt;li&gt;Looks it up in the database&lt;/li&gt;
&lt;li&gt;Creates a one-time reset token and saves it&lt;/li&gt;
&lt;li&gt;Sends you an email or SMS with a link or code&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;That's four steps, and some of them cost real money. Every SMS costs you something. Every email counts against your sending limits and your domain's reputation.&lt;/p&gt;

&lt;p&gt;So this endpoint is unusual. Most endpoints just &lt;em&gt;read&lt;/em&gt; data. This one &lt;em&gt;does things in the outside world&lt;/em&gt; on behalf of someone who isn't logged in.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why it's dangerous
&lt;/h3&gt;

&lt;p&gt;At 40,000 requests per second, a few different things could be going wrong at once:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Your bill.&lt;/strong&gt; If even a fraction of those requests send SMS, the cost adds up quickly. There's &lt;a href="https://www.twilio.com/docs/glossary/what-is-sms-pumping-fraud" rel="noopener noreferrer"&gt;a known fraud pattern&lt;/a&gt; called &lt;em&gt;SMS pumping&lt;/em&gt; (or toll fraud), where attackers trigger texts to premium numbers they profit from.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Your email reputation.&lt;/strong&gt; Email providers notice when you suddenly send huge volumes. Your real emails, like order receipts, might start landing in spam.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Your users.&lt;/strong&gt; Real people may get &lt;a href="https://cheatsheetseries.owasp.org/cheatsheets/Forgot_Password_Cheat_Sheet.html" rel="noopener noreferrer"&gt;flooded with reset emails&lt;/a&gt; they never asked for. That's confusing and a bit scary for them.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Account enumeration.&lt;/strong&gt; If your endpoint answers "email not found" for unknown addresses, the attacker can use it to &lt;a href="https://cheatsheetseries.owasp.org/cheatsheets/Forgot_Password_Cheat_Sheet.html" rel="noopener noreferrer"&gt;learn which emails have accounts&lt;/a&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Your system.&lt;/strong&gt; 40,000 database lookups and token writes per second can slow down everything else that shares that database, including logins and checkout.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  The business impact
&lt;/h3&gt;

&lt;p&gt;Put together, this isn't only a "server is busy" problem. It's money leaking, trust eroding, and possibly the whole product slowing down for everyone.&lt;/p&gt;

&lt;h3&gt;
  
  
  Reasonable assumptions
&lt;/h3&gt;

&lt;p&gt;Since the question gives us very little, I'll assume:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The app sits behind something like a load balancer or CDN (more on those below)&lt;/li&gt;
&lt;li&gt;Reset messages are sent through a third-party email or SMS provider&lt;/li&gt;
&lt;li&gt;We have basic logs and metrics, but no fancy bot protection yet&lt;/li&gt;
&lt;li&gt;The block was added somewhere sensible, like a firewall rule or app config&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If your setup is different, some of the reasoning below will shift. That's fine. The thinking process matters more than the exact tool.&lt;/p&gt;

&lt;h2&gt;
  
  
  So Why Didn't Blocking the IP Work?
&lt;/h2&gt;

&lt;p&gt;This is the heart of the question. The honest answer is "it depends," but there's a short list of usual suspects. Any one of them (or several together) explains what we're seeing.&lt;/p&gt;

&lt;p&gt;Here's an analogy I found helpful. Imagine a shop being flooded with prank phone calls. You block the number shown on caller ID. The calls keep coming. Why? Maybe the caller ID was fake. Maybe they switched phones. Maybe you blocked the number on the wrong phone line. Or maybe the calls &lt;em&gt;are&lt;/em&gt; blocked, but your phone still rings before the block kicks in.&lt;/p&gt;

&lt;p&gt;Each of those has a real-world equivalent.&lt;/p&gt;

&lt;h3&gt;
  
  
  1. The "IP" on your dashboard wasn't the real IP
&lt;/h3&gt;

&lt;p&gt;Most apps don't talk to users directly. Requests usually pass through a &lt;strong&gt;load balancer&lt;/strong&gt; (a traffic cop that spreads requests across your servers) or a &lt;strong&gt;CDN&lt;/strong&gt; (a network of servers around the world that sits in front of your app).&lt;/p&gt;

&lt;p&gt;Because of that, your app often learns the user's IP from an HTTP header called &lt;code&gt;X-Forwarded-For&lt;/code&gt;. The proxy adds it to say "this request originally came from X."&lt;/p&gt;

&lt;p&gt;Here's the catch: &lt;strong&gt;&lt;a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/X-Forwarded-For" rel="noopener noreferrer"&gt;a client can put anything it likes in that header&lt;/a&gt;.&lt;/strong&gt; If your app trusts it blindly, the attacker can write any IP they want.&lt;/p&gt;

&lt;p&gt;So a very plausible story is:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Your dashboard reads &lt;code&gt;X-Forwarded-For&lt;/code&gt; and shows one Indian IP&lt;/li&gt;
&lt;li&gt;Your firewall blocks that IP based on the &lt;em&gt;actual&lt;/em&gt; network connection&lt;/li&gt;
&lt;li&gt;The actual connections come from completely different addresses&lt;/li&gt;
&lt;li&gt;The block matches nothing, and the flood continues&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Or the reverse: you blocked the header value, and the attacker simply changed it.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. You blocked a proxy, not the attacker
&lt;/h3&gt;

&lt;p&gt;CDNs have servers in many countries, including India. If an attacker sits near a CDN server in Mumbai, all their traffic might arrive at your app &lt;em&gt;from that CDN server's IP&lt;/em&gt;.&lt;/p&gt;

&lt;p&gt;Block that IP at your origin, and two things happen. The CDN routes traffic through a different one of its servers, so the attack continues. And real users who happened to use that CDN server may get blocked too.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. The attacker just moved
&lt;/h3&gt;

&lt;p&gt;Renting lots of IP addresses is cheap. Attackers use &lt;strong&gt;botnets&lt;/strong&gt; (networks of hacked devices) and &lt;strong&gt;residential proxy services&lt;/strong&gt; (traffic routed through ordinary home internet connections). When one address gets blocked, the bot operator &lt;a href="https://blog.cloudflare.com/residential-proxy-bot-detection-using-machine-learning/" rel="noopener noreferrer"&gt;simply moves to new IP addresses&lt;/a&gt; and carries on.&lt;/p&gt;

&lt;p&gt;IPv6 makes this even easier. Home connections usually get a whole block of addresses, not just one. The &lt;a href="https://www.rfc-editor.org/rfc/rfc6177" rel="noopener noreferrer"&gt;IETF's guidance&lt;/a&gt; suggests something like a /56 for home users. Even a single /64 block contains about 18 quintillion addresses (2 to the power of 64). Blocking one address out of that barely matters.&lt;/p&gt;

&lt;h3&gt;
  
  
  4. The attacker skipped your front door
&lt;/h3&gt;

&lt;p&gt;If your block lives on the CDN or a &lt;strong&gt;WAF&lt;/strong&gt; (Web Application Firewall, a filter that inspects incoming requests), it only works for traffic that goes &lt;em&gt;through&lt;/em&gt; them.&lt;/p&gt;

&lt;p&gt;If your origin server's IP address &lt;a href="https://developers.cloudflare.com/fundamentals/concepts/cloudflare-ip-addresses/" rel="noopener noreferrer"&gt;has leaked&lt;/a&gt; (old DNS records, a subdomain that doesn't use the CDN, your mail server setup), the attacker can send requests straight to your server. Your shiny CDN rule never sees them.&lt;/p&gt;

&lt;h3&gt;
  
  
  5. The block was in the wrong layer, or didn't spread
&lt;/h3&gt;

&lt;p&gt;A few more boring but common causes:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The rule was added in app code, so every request still reaches your servers before being rejected&lt;/li&gt;
&lt;li&gt;You have servers in several regions, and the rule only went live in one&lt;/li&gt;
&lt;li&gt;Another rule earlier in the list allowed the traffic before the block rule ran&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  6. The block did work. The graph just looks the same.
&lt;/h3&gt;

&lt;p&gt;This one surprised me. &lt;strong&gt;Blocking an IP doesn't stop someone from sending requests.&lt;/strong&gt; It only changes how you respond.&lt;/p&gt;

&lt;p&gt;If your request-count graph is measured at the load balancer, it will still show 40,000 requests per second. They're just being rejected now. The real question isn't "did the number go down?" It's "are we still &lt;em&gt;sending emails, writing tokens, and hitting the database&lt;/em&gt;?"&lt;/p&gt;

&lt;p&gt;That distinction changes everything about how you respond.&lt;/p&gt;

&lt;h2&gt;
  
  
  The First 5 Minutes: Stop the Bleeding
&lt;/h2&gt;

&lt;p&gt;The goal right now is not to catch the attacker. It's not to find the perfect fix either. It's to &lt;strong&gt;stop the damage&lt;/strong&gt; so you have time to think.&lt;/p&gt;

&lt;p&gt;Think of a burst pipe. First you shut off the water. You figure out why the pipe burst later.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 1: Find out what's actually being harmed
&lt;/h3&gt;

&lt;p&gt;Remember point 6 above. Before doing anything else, check:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Are emails or SMS still going out at a high rate? (Your provider's dashboard usually shows this.)&lt;/li&gt;
&lt;li&gt;Is the database under pressure?&lt;/li&gt;
&lt;li&gt;Are other features, like login and checkout, getting slower?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This tells you whether you're in "money and users are being hurt" mode or "the graph looks scary but we're coping" mode.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 2: Protect the expensive part
&lt;/h3&gt;

&lt;p&gt;If messages are still being sent, that's the most urgent leak. Some fast options:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Pause or cap sending&lt;/strong&gt; in your email or SMS provider. Many providers let you set spending or volume limits.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Turn off SMS resets temporarily&lt;/strong&gt; with a feature flag (a switch that turns a feature on or off without redeploying code), and keep email only.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Stop sending duplicates.&lt;/strong&gt; If the same account already got a reset message in the last few minutes, don't send another.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The trade-off: a few real users might not get their reset message for a while. That's annoying, but it's much better than a five-figure SMS bill or your whole email domain getting flagged as spam.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 3: Put a hard limit on the endpoint itself
&lt;/h3&gt;

&lt;p&gt;Add a blunt &lt;strong&gt;rate limit&lt;/strong&gt; on &lt;code&gt;/reset-password&lt;/code&gt; as a whole. A rate limiter is simply a rule like "no more than N requests per second for this endpoint." Anything over that gets rejected quickly, with an &lt;a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/429" rel="noopener noreferrer"&gt;HTTP 429&lt;/a&gt; ("Too Many Requests") response.&lt;/p&gt;

&lt;p&gt;Pick a number well above your normal traffic, but far below 40,000. Some real users will be caught in it. That's the cost of stabilizing.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 4: Tell people
&lt;/h3&gt;

&lt;p&gt;Post a quick note in your incident channel. Let support know that users might mention odd reset emails. Communication feels like a soft step, but it prevents five people from "fixing" the same thing in five different ways.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Next 15 Minutes: Get Smarter About Who You Block
&lt;/h2&gt;

&lt;p&gt;Now that the bleeding has slowed, you can move from blunt tools to sharper ones. The big shift here is this:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Stop asking "which IP is attacking?" and start asking "what does this attack have in common?"&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h3&gt;
  
  
  Fix how you read the client IP
&lt;/h3&gt;

&lt;p&gt;Make sure your app only trusts &lt;code&gt;X-Forwarded-For&lt;/code&gt; values added by &lt;strong&gt;your own&lt;/strong&gt; load balancer or CDN, not whatever the client sent. &lt;a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/X-Forwarded-For" rel="noopener noreferrer"&gt;MDN describes two ways to do this&lt;/a&gt;: tell your app how many proxies sit in front of it, or give it a list of your proxies' addresses. Either way, it reads the header from the right-hand end and ignores what the client wrote.&lt;/p&gt;

&lt;p&gt;Once that's right, your dashboard shows the &lt;em&gt;real&lt;/em&gt; source addresses. You may suddenly see that "one IP in India" is really thousands of IPs all over the world. That alone often explains the whole mystery.&lt;/p&gt;

&lt;h3&gt;
  
  
  Rate limit by the thing being attacked, not just by IP
&lt;/h3&gt;

&lt;p&gt;This was the biggest "aha" for me. An attacker can change their IP easily. They can't easily change &lt;strong&gt;which accounts they're targeting&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;So add limits like:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Per email or phone number:&lt;/strong&gt; at most a few reset requests per hour for the same account (&lt;a href="https://cheatsheetseries.owasp.org/cheatsheets/Forgot_Password_Cheat_Sheet.html" rel="noopener noreferrer"&gt;OWASP recommends&lt;/a&gt; this kind of per-account limit)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Per IP and per network range:&lt;/strong&gt; limit a whole range (say, an IPv6 /64) instead of single addresses&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Global:&lt;/strong&gt; a ceiling for the endpoint as a whole, like the one from Step 3&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Each layer catches something the others miss. Per-account limits protect your users' inboxes. Range limits catch rotating addresses. The global limit protects your system no matter what.&lt;/p&gt;

&lt;p&gt;The trade-off: per-account limits can be abused too. Someone could deliberately use up the limit for &lt;em&gt;your&lt;/em&gt; email, so you can't reset your own password for an hour. Keep the window short and the limit sensible.&lt;/p&gt;

&lt;h3&gt;
  
  
  Move the block to the edge
&lt;/h3&gt;

&lt;p&gt;The "edge" means the outermost layer of your system, usually a CDN or WAF. Blocking there means bad requests never reach your servers at all, so they cost you almost nothing.&lt;/p&gt;

&lt;p&gt;If your origin's IP might be exposed, &lt;a href="https://developers.cloudflare.com/fundamentals/concepts/cloudflare-ip-addresses/" rel="noopener noreferrer"&gt;lock it down&lt;/a&gt; so it &lt;strong&gt;only accepts traffic from your CDN&lt;/strong&gt;. Then nobody can sneak around the front door.&lt;/p&gt;

&lt;h3&gt;
  
  
  Look for a fingerprint
&lt;/h3&gt;

&lt;p&gt;Attack traffic usually shares something. Common examples:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The same or a strange &lt;strong&gt;User-Agent&lt;/strong&gt; (the text a browser sends to say what it is)&lt;/li&gt;
&lt;li&gt;Missing headers that real browsers always send&lt;/li&gt;
&lt;li&gt;The same request body pattern, or emails following a pattern like &lt;code&gt;user1@&lt;/code&gt;, &lt;code&gt;user2@&lt;/code&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If you find one, block or challenge that pattern at the edge. It works no matter how many IPs the attacker uses.&lt;/p&gt;

&lt;h3&gt;
  
  
  Add a challenge, not just a wall
&lt;/h3&gt;

&lt;p&gt;A &lt;strong&gt;CAPTCHA&lt;/strong&gt; or an invisible bot check asks the client to prove it's a real browser with a real human before the reset goes through. Real users barely notice. Scripts sending 40,000 requests a second struggle.&lt;/p&gt;

&lt;p&gt;The trade-off is accessibility and friction. &lt;a href="https://www.w3.org/TR/turingtest/" rel="noopener noreferrer"&gt;Some CAPTCHAs are hard for people with disabilities&lt;/a&gt;, including people using screen readers, and some users just find them annoying. Many teams only show the challenge when traffic looks suspicious, rather than to everyone, all the time.&lt;/p&gt;

&lt;h3&gt;
  
  
  Should you block all of India?
&lt;/h3&gt;

&lt;p&gt;It's tempting. Please don't, unless you truly have no users there. You'd be locking out a huge number of real people to stop one attacker who, as we've seen, may not even be in India. &lt;a href="https://blog.cloudflare.com/residential-proxy-bot-detection-using-machine-learning/" rel="noopener noreferrer"&gt;Well-funded bots can get around country blocks anyway&lt;/a&gt;. Geography-based blocks are a last resort, not a first move.&lt;/p&gt;

&lt;h2&gt;
  
  
  Root Cause Investigation: Once Things Are Calm
&lt;/h2&gt;

&lt;p&gt;With the system stable, you can finally play detective. Here's where I'd look, and what each place can tell you.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Where to look&lt;/th&gt;
&lt;th&gt;What you're trying to learn&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Edge / CDN / WAF logs&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;The real source IPs, countries, and networks. Is it one IP or thousands?&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;App logs&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;What's in &lt;code&gt;X-Forwarded-For&lt;/code&gt; vs. the actual connection IP. Do they match?&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Request details&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Shared User-Agents, headers, or patterns in the emails being targeted&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Metrics&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;When it started, how it grew, and whether it changed when you blocked the IP&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Tracing&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Which step is slow or expensive: the DB lookup, token write, or message send&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Email / SMS provider&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;How many messages actually went out, to which countries, and what it cost&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Database health&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Connection pool usage, slow queries, lock contention&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Recent deployments&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Did a recent change remove a rate limit or a CAPTCHA?&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Infrastructure changes&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;New subdomains, DNS changes, or anything that might expose the origin IP&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;A quick note on &lt;strong&gt;tracing&lt;/strong&gt;, since it's a newer term for many people. Tracing follows one single request through every part of your system, like a parcel tracking number. It shows exactly where time and effort were spent.&lt;/p&gt;

&lt;p&gt;A few questions I'd want answered by the end:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Was the "single IP" ever real, or a header value?&lt;/li&gt;
&lt;li&gt;Were the targeted emails real users, random guesses, or premium-rate phone numbers?&lt;/li&gt;
&lt;li&gt;Did the attacker get anything useful, like a list of which emails have accounts?&lt;/li&gt;
&lt;li&gt;How much did the incident cost in messages sent?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The answers decide which long-term fixes matter most for &lt;em&gt;your&lt;/em&gt; system.&lt;/p&gt;

&lt;h2&gt;
  
  
  Long-Term Prevention: Design the Endpoint to Be Boring
&lt;/h2&gt;

&lt;p&gt;The best outcome is a reset endpoint that's simply &lt;em&gt;not worth attacking&lt;/em&gt;. Each idea below removes some reason an attacker would bother.&lt;/p&gt;

&lt;h3&gt;
  
  
  Give the same answer every time
&lt;/h3&gt;

&lt;p&gt;Whether the email exists or not, &lt;a href="https://cheatsheetseries.owasp.org/cheatsheets/Forgot_Password_Cheat_Sheet.html" rel="noopener noreferrer"&gt;respond with the same message&lt;/a&gt;, like "If an account exists, we've sent a reset link." Try to make the response take roughly the same time too. OWASP suggests sending the email in the background, which also helps with the next point.&lt;/p&gt;

&lt;p&gt;This kills account enumeration. The attacker learns nothing from your replies.&lt;/p&gt;

&lt;h3&gt;
  
  
  Make the request cheap, and the sending asynchronous
&lt;/h3&gt;

&lt;p&gt;Don't send the email &lt;em&gt;during&lt;/em&gt; the request. Instead, put a small job on a &lt;strong&gt;queue&lt;/strong&gt; (a waiting list that background workers process at their own pace) and reply right away.&lt;/p&gt;

&lt;p&gt;Why it helps: the queue acts like a shock absorber. Even if 40,000 requests arrive, the workers only send as fast as you allow. You can also dedupe ("this account already has a pending reset") before anything goes out.&lt;/p&gt;

&lt;h3&gt;
  
  
  Layered rate limiting
&lt;/h3&gt;

&lt;p&gt;Keep the layers from earlier as permanent settings: per account, per IP range, and global. Store the counters somewhere fast and shared, like &lt;strong&gt;Redis&lt;/strong&gt; (an in-memory data store that's very quick at counting things), so every server sees the same numbers.&lt;/p&gt;

&lt;h3&gt;
  
  
  Bot protection at the edge
&lt;/h3&gt;

&lt;p&gt;Most CDNs and cloud providers offer WAF rules and bot detection. They can challenge suspicious traffic before it reaches you. This is usually much cheaper than building your own.&lt;/p&gt;

&lt;h3&gt;
  
  
  Protect the origin
&lt;/h3&gt;

&lt;p&gt;Make sure your servers only accept traffic from your CDN or load balancer. Don't leave a side door open.&lt;/p&gt;

&lt;h3&gt;
  
  
  Circuit breaker on your message provider
&lt;/h3&gt;

&lt;p&gt;A &lt;strong&gt;circuit breaker&lt;/strong&gt; is &lt;a href="https://martinfowler.com/bliki/CircuitBreaker.html" rel="noopener noreferrer"&gt;like the one in your house&lt;/a&gt;. If something goes badly wrong, it trips and stops the flow automatically.&lt;/p&gt;

&lt;p&gt;Here, it could mean: "if we've sent more than X reset messages in 10 minutes, pause sending and alert a human." That caps your worst-case bill even if every other defense fails.&lt;/p&gt;

&lt;h3&gt;
  
  
  Restrict where SMS can go
&lt;/h3&gt;

&lt;p&gt;If you only serve certain countries, only allow SMS to those countries. &lt;a href="https://www.twilio.com/docs/glossary/what-is-sms-pumping-fraud" rel="noopener noreferrer"&gt;Twilio, for example, recommends&lt;/a&gt; turning off every country you don't plan to send messages to. It's one of the simplest defenses against SMS pumping.&lt;/p&gt;

&lt;h3&gt;
  
  
  Feature flags for emergency switches
&lt;/h3&gt;

&lt;p&gt;Have ready-made switches for things like "disable SMS reset" or "require CAPTCHA for everyone." During an incident you want to flip a switch, not write and deploy code at 2 AM.&lt;/p&gt;

&lt;h3&gt;
  
  
  Monitoring and alerts on the right signals
&lt;/h3&gt;

&lt;p&gt;Alert on the things that actually hurt, not just request count:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Reset messages sent per minute&lt;/li&gt;
&lt;li&gt;SMS spend per hour&lt;/li&gt;
&lt;li&gt;Reset requests per account&lt;/li&gt;
&lt;li&gt;Share of reset requests for emails that don't exist&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That last one is a great early warning. Real users mostly type real emails. Attackers guessing don't.&lt;/p&gt;

&lt;h3&gt;
  
  
  Load testing and practice
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;Load testing&lt;/strong&gt; means sending fake heavy traffic to your system on purpose, in a safe environment, to see where it breaks. Do it for this endpoint specifically. Then run a practice incident ("game day") so the team knows where the switches are before they need them.&lt;/p&gt;

&lt;h3&gt;
  
  
  What about auto scaling?
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;Auto scaling&lt;/strong&gt; means automatically adding more servers when traffic grows. It's great for real popularity. For an attack, it can backfire: you scale up and simply pay more to process junk. Scaling buys you time. It doesn't solve this problem.&lt;/p&gt;

&lt;h2&gt;
  
  
  Common Mistakes
&lt;/h2&gt;

&lt;p&gt;These are the traps I'd most likely have fallen into myself before working through this.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Treating the IP as an identity.&lt;/strong&gt; An IP address is more like a temporary seat on a bus than a name tag. Many people can share one, and one person can switch between many.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Trusting &lt;code&gt;X-Forwarded-For&lt;/code&gt; blindly.&lt;/strong&gt; It's just text the client can write. Only trust values added by proxies you control. &lt;a href="https://www.stackhawk.com/blog/do-you-trust-your-x-forwarded-for-header/" rel="noopener noreferrer"&gt;StackHawk wrote about&lt;/a&gt; how this exact mistake let spoofed IPs get past their own API's rate limiting.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Playing whack-a-mole.&lt;/strong&gt; Blocking IPs one by one feels productive, but attackers can rotate faster than you can type. Look for patterns instead.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Measuring the wrong thing.&lt;/strong&gt; "Requests per second" doesn't tell you if you're losing money. "Messages sent" and "cost per hour" do.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Sending messages inside the request.&lt;/strong&gt; It ties your cost directly to the attacker's speed. A queue breaks that link.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Blocking a whole country.&lt;/strong&gt; It punishes real users and often doesn't even hit the attacker.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Scaling up to absorb the attack.&lt;/strong&gt; You end up paying more to serve the attacker better.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Revealing whether an account exists.&lt;/strong&gt; Different messages for "found" and "not found" turn your reset page into a free lookup tool.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Forgetting real users in the response.&lt;/strong&gt; Overly harsh limits or CAPTCHAs can lock out the very people the reset page exists to help.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Key Takeaways
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The IP you see may not be the attacker's IP.&lt;/strong&gt; Headers can be faked, proxies hide the real source, and attackers rotate addresses cheaply.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Blocking doesn't stop sending.&lt;/strong&gt; It only changes your response. Measure harm (messages, cost, DB load), not just traffic.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Stabilize first, investigate second.&lt;/strong&gt; Cap the expensive part, add a global limit, then get clever.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Rate limit by what's being targeted.&lt;/strong&gt; Per-account limits survive IP rotation in a way per-IP limits can't.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Push defenses to the edge&lt;/strong&gt; and make sure nobody can bypass it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Design the endpoint to be boring:&lt;/strong&gt; same response every time, queued sending, hard caps on spend.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Every defense has a cost to real users.&lt;/strong&gt; Good incident response keeps them in mind the whole time.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If there's one sentence I'll remember from this, it's: &lt;em&gt;don't fight the address, fight the pattern.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Over to You
&lt;/h2&gt;

&lt;p&gt;I'm still learning this space, so I'd genuinely like to hear from people who've dealt with it for real.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;If you've handled abuse on a password reset or OTP endpoint, what was the one defense that actually made the difference, and which "obvious" fix turned out to be useless?&lt;/strong&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Sources and Further Reading
&lt;/h2&gt;

&lt;p&gt;These are the pages I leaned on while working through this. If you want to go deeper, start with the first two.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;a href="https://cheatsheetseries.owasp.org/cheatsheets/Forgot_Password_Cheat_Sheet.html" rel="noopener noreferrer"&gt;OWASP Forgot Password Cheat Sheet&lt;/a&gt;: consistent responses, user enumeration, per-account limits, and flooding&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/X-Forwarded-For" rel="noopener noreferrer"&gt;MDN: X-Forwarded-For header&lt;/a&gt;: why the header can be spoofed and how to pick a trustworthy IP&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://www.stackhawk.com/blog/do-you-trust-your-x-forwarded-for-header/" rel="noopener noreferrer"&gt;StackHawk: Do You Trust Your X-Forwarded-For Header?&lt;/a&gt;: a real story of spoofed IPs bypassing rate limiting&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://blog.cloudflare.com/residential-proxy-bot-detection-using-machine-learning/" rel="noopener noreferrer"&gt;Cloudflare: Using machine learning to detect bot attacks that leverage residential proxies&lt;/a&gt;: how residential proxies get around IP blocks, country blocks, and rate limits&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://developers.cloudflare.com/fundamentals/concepts/cloudflare-ip-addresses/" rel="noopener noreferrer"&gt;Cloudflare docs: Cloudflare IP addresses&lt;/a&gt;: how an exposed origin IP lets attackers bypass the CDN, and allowlisting&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://www.rfc-editor.org/rfc/rfc6177" rel="noopener noreferrer"&gt;RFC 6177: IPv6 Address Assignment to End Sites&lt;/a&gt;: how much IPv6 address space home users typically get&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://www.twilio.com/docs/glossary/what-is-sms-pumping-fraud" rel="noopener noreferrer"&gt;Twilio: What is SMS Pumping Fraud?&lt;/a&gt;: how the fraud works and the geo-permission defense&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/429" rel="noopener noreferrer"&gt;MDN: 429 Too Many Requests&lt;/a&gt;: the status code used for rate limiting&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://martinfowler.com/bliki/CircuitBreaker.html" rel="noopener noreferrer"&gt;Martin Fowler: Circuit Breaker&lt;/a&gt;: the classic explanation of the pattern&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://www.w3.org/TR/turingtest/" rel="noopener noreferrer"&gt;W3C: Inaccessibility of CAPTCHA&lt;/a&gt;: accessibility trade-offs of bot challenges&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The incident steps and their order (stabilize first, then target, then investigate) are my own reasoning, not taken from one source.&lt;/p&gt;

</description>
      <category>security</category>
      <category>webdev</category>
      <category>systemdesign</category>
      <category>devops</category>
    </item>
    <item>
      <title>Login System Meltdown: Handling a 40 Traffic Spike</title>
      <dc:creator>Nachiket Gholap</dc:creator>
      <pubDate>Thu, 06 Aug 2026 18:14:19 +0000</pubDate>
      <link>https://dev.to/nachiketgholap/login-system-meltdown-handling-a-40x-traffic-spike-2ldk</link>
      <guid>https://dev.to/nachiketgholap/login-system-meltdown-handling-a-40x-traffic-spike-2ldk</guid>
      <description>&lt;h2&gt;
  
  
  I Have 15 Minutes to Save a Login System. Here's What I'd Do.
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Scenario&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Your authentication service normally handles 500 login requests per second. Suddenly, traffic spikes to 20,000 login requests per second. The CPU is stuck at 100%. Users can't log in. Customer support is flooded. Your manager asks, "Can you fix it?" You have 15 minutes.&lt;/p&gt;

&lt;p&gt;What do you do?&lt;/p&gt;

&lt;p&gt;Let's walk through this like we're the engineer on call.&lt;/p&gt;

&lt;h2&gt;
  
  
  Before Touching Anything, Understand the Problem
&lt;/h2&gt;

&lt;p&gt;The biggest mistake during production incidents is assuming you already know the cause.&lt;/p&gt;

&lt;p&gt;A login failure can happen because of dozens of reasons.&lt;/p&gt;

&lt;p&gt;The CPU hitting 100% is only a symptom.&lt;/p&gt;

&lt;p&gt;The real problem could be:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A sudden marketing campaign&lt;/li&gt;
&lt;li&gt;A celebrity mentioning your app&lt;/li&gt;
&lt;li&gt;A bot attack&lt;/li&gt;
&lt;li&gt;Credential stuffing attack&lt;/li&gt;
&lt;li&gt;A DDoS attack&lt;/li&gt;
&lt;li&gt;Database slowdown&lt;/li&gt;
&lt;li&gt;Cache outage&lt;/li&gt;
&lt;li&gt;Recent deployment bug&lt;/li&gt;
&lt;li&gt;Third-party authentication provider failure&lt;/li&gt;
&lt;li&gt;Infrastructure misconfiguration&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If you immediately restart the service without understanding what's happening, you might temporarily reduce CPU usage but make recovery even harder.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The first goal is not fixing the root cause.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The first goal is &lt;strong&gt;stopping the bleeding&lt;/strong&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  What Is Happening?
&lt;/h2&gt;

&lt;p&gt;Let's compare the numbers.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Metric&lt;/th&gt;
&lt;th&gt;Normal State&lt;/th&gt;
&lt;th&gt;Incident State&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Traffic&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;500 logins/sec&lt;/td&gt;
&lt;td&gt;20,000 logins/sec&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;CPU Usage&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Healthy&lt;/td&gt;
&lt;td&gt;100%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;User Impact&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Users login normally&lt;/td&gt;
&lt;td&gt;Users can't login&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;That's a &lt;strong&gt;40x traffic spike&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Most authentication services are designed for predictable traffic.&lt;/p&gt;

&lt;p&gt;Receiving forty times the normal load overwhelms nearly every part of the system:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;CPU&lt;/li&gt;
&lt;li&gt;Memory&lt;/li&gt;
&lt;li&gt;Database&lt;/li&gt;
&lt;li&gt;Network&lt;/li&gt;
&lt;li&gt;Session storage&lt;/li&gt;
&lt;li&gt;Cache&lt;/li&gt;
&lt;li&gt;External identity providers&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Even if only one component becomes overloaded, the entire login flow can fail.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why Is This Dangerous?
&lt;/h2&gt;

&lt;p&gt;When users cannot log in:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Customers cannot access the application.&lt;/li&gt;
&lt;li&gt;Revenue may stop.&lt;/li&gt;
&lt;li&gt;Support tickets increase rapidly.&lt;/li&gt;
&lt;li&gt;Frustration spreads on social media.&lt;/li&gt;
&lt;li&gt;Business reputation suffers.&lt;/li&gt;
&lt;li&gt;Engineers are forced into emergency response mode.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For many companies, the login service is the front door.&lt;/p&gt;

&lt;p&gt;If people cannot enter, nothing else matters.&lt;/p&gt;

&lt;h2&gt;
  
  
  Reasonable Assumptions
&lt;/h2&gt;

&lt;p&gt;Since we only have the scenario, let's make a few reasonable assumptions:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The application was working normally before.&lt;/li&gt;
&lt;li&gt;Infrastructure monitoring is available.&lt;/li&gt;
&lt;li&gt;Logs and metrics exist.&lt;/li&gt;
&lt;li&gt;The production team can scale infrastructure if necessary.&lt;/li&gt;
&lt;li&gt;The authentication service is stateless.&lt;/li&gt;
&lt;li&gt;The database stores user credentials.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If any of these assumptions are false, the response may differ.&lt;/p&gt;

&lt;h2&gt;
  
  
  The First 5 Minutes: Stabilize the System
&lt;/h2&gt;

&lt;p&gt;Notice I didn't say "fix the problem."&lt;/p&gt;

&lt;p&gt;At this stage, stability matters more than perfection.&lt;/p&gt;

&lt;h3&gt;
  
  
  1. Declare an Incident
&lt;/h3&gt;

&lt;p&gt;The first action isn't technical.&lt;/p&gt;

&lt;p&gt;It's operational.&lt;/p&gt;

&lt;p&gt;Notify the team immediately.&lt;/p&gt;

&lt;p&gt;Bring in:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Backend engineers&lt;/li&gt;
&lt;li&gt;DevOps/SRE&lt;/li&gt;
&lt;li&gt;Database administrator (if needed)&lt;/li&gt;
&lt;li&gt;Security team (if an attack is suspected)&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Production incidents are team sports.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. Check Monitoring Dashboards
&lt;/h3&gt;

&lt;p&gt;Look at:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;CPU&lt;/li&gt;
&lt;li&gt;Memory&lt;/li&gt;
&lt;li&gt;Network traffic&lt;/li&gt;
&lt;li&gt;Error rate&lt;/li&gt;
&lt;li&gt;Response time&lt;/li&gt;
&lt;li&gt;Request rate&lt;/li&gt;
&lt;li&gt;Database connections&lt;/li&gt;
&lt;li&gt;Cache hit ratio&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;You're trying to answer:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Where is the bottleneck?&lt;/strong&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  3. Verify Whether the Traffic Is Legitimate
&lt;/h3&gt;

&lt;p&gt;Ask questions like:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Did marketing launch a campaign?&lt;/li&gt;
&lt;li&gt;Did a mobile app update just release?&lt;/li&gt;
&lt;li&gt;Did a popular influencer mention us?&lt;/li&gt;
&lt;li&gt;Is a partner integrating with our API?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Or...&lt;/p&gt;

&lt;p&gt;Is someone attacking us?&lt;/p&gt;

&lt;p&gt;This changes everything.&lt;/p&gt;

&lt;h3&gt;
  
  
  4. Protect the Login Service
&lt;/h3&gt;

&lt;p&gt;If traffic is overwhelming the system, prevent it from getting worse.&lt;/p&gt;

&lt;p&gt;Possible actions:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Enable rate limiting.&lt;/li&gt;
&lt;li&gt;Block abusive IP addresses.&lt;/li&gt;
&lt;li&gt;Enable Web Application Firewall (WAF) rules.&lt;/li&gt;
&lt;li&gt;Activate bot protection.&lt;/li&gt;
&lt;li&gt;Reject excessive requests.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The goal is to preserve capacity for genuine users.&lt;/p&gt;

&lt;p&gt;Think of it like crowd control at a stadium.&lt;/p&gt;

&lt;p&gt;Instead of allowing everyone to rush the entrance at once, security lets people enter in an orderly way.&lt;/p&gt;

&lt;h3&gt;
  
  
  5. Scale the Authentication Service
&lt;/h3&gt;

&lt;p&gt;If the traffic is legitimate and the application supports horizontal scaling:&lt;/p&gt;

&lt;p&gt;Increase the number of authentication service instances.&lt;/p&gt;

&lt;p&gt;More servers mean the incoming requests are shared across multiple machines instead of one overloaded server.&lt;/p&gt;

&lt;p&gt;However...&lt;/p&gt;

&lt;p&gt;Scaling isn't always enough.&lt;/p&gt;

&lt;p&gt;If every new server still queries the same overloaded database, you've simply moved the bottleneck.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Next 15 Minutes: Tactical Actions
&lt;/h2&gt;

&lt;p&gt;Once the immediate pressure is under control, start investigating more deeply.&lt;/p&gt;

&lt;h3&gt;
  
  
  Check Error Logs
&lt;/h3&gt;

&lt;p&gt;Look for patterns.&lt;/p&gt;

&lt;p&gt;Examples:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Database timeout&lt;/li&gt;
&lt;li&gt;Authentication timeout&lt;/li&gt;
&lt;li&gt;Connection pool exhausted&lt;/li&gt;
&lt;li&gt;Out of memory&lt;/li&gt;
&lt;li&gt;Thread exhaustion&lt;/li&gt;
&lt;li&gt;SSL failures&lt;/li&gt;
&lt;li&gt;Third-party authentication errors&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Logs often reveal whether failures are occurring before, during, or after authentication.&lt;/p&gt;

&lt;h3&gt;
  
  
  Examine Database Health
&lt;/h3&gt;

&lt;p&gt;Authentication usually depends on a database.&lt;/p&gt;

&lt;p&gt;Check:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;CPU&lt;/li&gt;
&lt;li&gt;Active connections&lt;/li&gt;
&lt;li&gt;Slow queries&lt;/li&gt;
&lt;li&gt;Locks&lt;/li&gt;
&lt;li&gt;Replication lag&lt;/li&gt;
&lt;li&gt;Disk usage&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If the database is overloaded, adding more application servers won't help.&lt;/p&gt;

&lt;h3&gt;
  
  
  Check the Cache
&lt;/h3&gt;

&lt;p&gt;Many authentication systems cache:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Session data&lt;/li&gt;
&lt;li&gt;User profiles&lt;/li&gt;
&lt;li&gt;Tokens&lt;/li&gt;
&lt;li&gt;Permissions&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If the cache fails, every login request hits the database directly.&lt;/p&gt;

&lt;p&gt;Imagine a library where the index cards disappear.&lt;/p&gt;

&lt;p&gt;Now everyone has to search every shelf manually.&lt;/p&gt;

&lt;p&gt;That's what happens when a cache goes down.&lt;/p&gt;

&lt;h3&gt;
  
  
  Check Recent Deployments
&lt;/h3&gt;

&lt;p&gt;Ask:&lt;/p&gt;

&lt;p&gt;"What changed today?"&lt;/p&gt;

&lt;p&gt;Many production incidents begin immediately after:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;New release&lt;/li&gt;
&lt;li&gt;Configuration change&lt;/li&gt;
&lt;li&gt;Feature flag enabled&lt;/li&gt;
&lt;li&gt;Infrastructure update&lt;/li&gt;
&lt;li&gt;Database migration&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If the timing matches, rolling back the deployment may be the fastest recovery option.&lt;/p&gt;

&lt;h3&gt;
  
  
  Investigate Third-Party Dependencies
&lt;/h3&gt;

&lt;p&gt;Modern authentication often relies on external services.&lt;/p&gt;

&lt;p&gt;Examples:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;OAuth providers&lt;/li&gt;
&lt;li&gt;Identity providers&lt;/li&gt;
&lt;li&gt;Multi-factor authentication services&lt;/li&gt;
&lt;li&gt;Email or SMS providers&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If one of these is unavailable, login failures may not be your fault.&lt;/p&gt;

&lt;h3&gt;
  
  
  What If This Is a Security Attack?
&lt;/h3&gt;

&lt;p&gt;A sudden 40x increase in login attempts could indicate:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Credential stuffing&lt;/li&gt;
&lt;li&gt;Password spraying&lt;/li&gt;
&lt;li&gt;Bot attack&lt;/li&gt;
&lt;li&gt;DDoS attack&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;In that case:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Notify the security team.&lt;/li&gt;
&lt;li&gt;Block malicious traffic.&lt;/li&gt;
&lt;li&gt;Enable CAPTCHA where appropriate.&lt;/li&gt;
&lt;li&gt;Tighten rate limits.&lt;/li&gt;
&lt;li&gt;Monitor suspicious IP ranges.&lt;/li&gt;
&lt;li&gt;Protect user accounts from brute-force attempts.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Treating an attack as normal traffic can make the situation much worse.&lt;/p&gt;

&lt;h3&gt;
  
  
  Common Technical Techniques That Help
&lt;/h3&gt;

&lt;h4&gt;
  
  
  Rate Limiting
&lt;/h4&gt;

&lt;p&gt;Limits how many login requests a user or IP address can make within a certain time.&lt;/p&gt;

&lt;p&gt;This prevents a few bad actors from consuming all available resources.&lt;/p&gt;

&lt;h4&gt;
  
  
  Load Balancing
&lt;/h4&gt;

&lt;p&gt;A load balancer distributes incoming requests across multiple application servers.&lt;/p&gt;

&lt;p&gt;Instead of one server handling everything, the work is shared.&lt;/p&gt;

&lt;p&gt;Think of it like opening more checkout counters in a supermarket.&lt;/p&gt;

&lt;h4&gt;
  
  
  Auto Scaling
&lt;/h4&gt;

&lt;p&gt;Cloud platforms can automatically create additional servers when CPU usage or traffic exceeds predefined thresholds.&lt;/p&gt;

&lt;p&gt;This helps absorb sudden traffic spikes without manual intervention.&lt;/p&gt;

&lt;h4&gt;
  
  
  Caching
&lt;/h4&gt;

&lt;p&gt;Store frequently accessed information in fast memory instead of repeatedly querying the database.&lt;/p&gt;

&lt;p&gt;This reduces database load and improves response times.&lt;/p&gt;

&lt;h4&gt;
  
  
  Circuit Breakers
&lt;/h4&gt;

&lt;p&gt;If a downstream service becomes slow or unavailable, a circuit breaker temporarily stops sending requests to it.&lt;/p&gt;

&lt;p&gt;This prevents failures from cascading through the entire system.&lt;/p&gt;

&lt;h3&gt;
  
  
  Queueing Non-Critical Work
&lt;/h3&gt;

&lt;p&gt;Some login-related tasks don't need to happen immediately.&lt;/p&gt;

&lt;p&gt;For example:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Audit logging&lt;/li&gt;
&lt;li&gt;Analytics events&lt;/li&gt;
&lt;li&gt;Welcome emails&lt;/li&gt;
&lt;li&gt;Notification updates&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Move these to asynchronous queues so the login request can complete faster.&lt;/p&gt;

&lt;h3&gt;
  
  
  Root Cause Investigation
&lt;/h3&gt;

&lt;p&gt;After users can log in again, begin the detailed investigation.&lt;/p&gt;

&lt;p&gt;Review:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Application logs&lt;/li&gt;
&lt;li&gt;Infrastructure metrics&lt;/li&gt;
&lt;li&gt;Distributed tracing&lt;/li&gt;
&lt;li&gt;Database performance&lt;/li&gt;
&lt;li&gt;Cache metrics&lt;/li&gt;
&lt;li&gt;Recent deployments&lt;/li&gt;
&lt;li&gt;Network changes&lt;/li&gt;
&lt;li&gt;Security alerts&lt;/li&gt;
&lt;li&gt;Third-party service status&lt;/li&gt;
&lt;li&gt;Capacity planning reports&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Ask:&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Why did this happen?&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Then ask an even better question:&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Why wasn't the system prepared for it?&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;That's where the biggest learning happens.&lt;/p&gt;

&lt;h3&gt;
  
  
  Long-Term Prevention
&lt;/h3&gt;

&lt;p&gt;A mature engineering team doesn't stop after restoring service.&lt;/p&gt;

&lt;p&gt;They improve the system so the same incident is less likely to happen again.&lt;/p&gt;

&lt;h4&gt;
  
  
  Capacity Planning
&lt;/h4&gt;

&lt;p&gt;Estimate future traffic growth and ensure infrastructure can handle expected peaks.&lt;/p&gt;

&lt;h4&gt;
  
  
  Load Testing
&lt;/h4&gt;

&lt;p&gt;Simulate realistic traffic before releasing changes.&lt;/p&gt;

&lt;p&gt;Don't assume the system can handle high load. Prove it.&lt;/p&gt;

&lt;h4&gt;
  
  
  Auto Scaling Policies
&lt;/h4&gt;

&lt;p&gt;Configure automatic scaling based on CPU usage, request rate, or latency instead of relying on manual intervention.&lt;/p&gt;

&lt;h4&gt;
  
  
  Better Observability
&lt;/h4&gt;

&lt;p&gt;Invest in:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Dashboards&lt;/li&gt;
&lt;li&gt;Alerts&lt;/li&gt;
&lt;li&gt;Logs&lt;/li&gt;
&lt;li&gt;Metrics&lt;/li&gt;
&lt;li&gt;Distributed tracing&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Good observability helps engineers detect problems before customers report them.&lt;/p&gt;

&lt;h4&gt;
  
  
  Smarter Rate Limiting
&lt;/h4&gt;

&lt;p&gt;Apply limits based on users, IP addresses, geographic regions, or API keys to protect the system without blocking legitimate traffic.&lt;/p&gt;

&lt;h4&gt;
  
  
  Database Optimization
&lt;/h4&gt;

&lt;p&gt;Reduce unnecessary queries, add indexes where appropriate, optimize slow queries, and consider read replicas if read traffic becomes a bottleneck.&lt;/p&gt;

&lt;h4&gt;
  
  
  Disaster Recovery Drills
&lt;/h4&gt;

&lt;p&gt;Practice incident response regularly.&lt;/p&gt;

&lt;p&gt;Teams that rehearse outages respond faster and with fewer mistakes when real incidents occur.&lt;/p&gt;

&lt;h3&gt;
  
  
  Common Mistakes During Incidents
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;Restarting services without understanding the problem.&lt;/li&gt;
&lt;li&gt;Scaling application servers while ignoring an overloaded database.&lt;/li&gt;
&lt;li&gt;Focusing only on CPU instead of end-to-end system health.&lt;/li&gt;
&lt;li&gt;Ignoring the possibility of a security attack.&lt;/li&gt;
&lt;li&gt;Making multiple production changes simultaneously, making it impossible to identify what helped or hurt.&lt;/li&gt;
&lt;li&gt;Failing to communicate with stakeholders and customers during the outage.&lt;/li&gt;
&lt;li&gt;Skipping the post-incident review after recovery.
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Traffic Spike Detected
       │
       ▼
Is it legitimate traffic?
       │
 ┌─────┴─────┐
 │           │
Yes          No
 │           │
 ▼           ▼
Scale System   Block Malicious Traffic
 │           │
 └─────┬─────┘
       │
       ▼
System Stabilized?
       │
 ┌─────┴─────┐
 │           │
No          Yes
 │           │
 ▼           ▼
Find Bottleneck  Root Cause Analysis
       │
       ▼
Implement Long-Term Fixes
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Key Takeaways
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;The first priority during an incident is stabilization, not perfection.&lt;/li&gt;
&lt;li&gt;A CPU spike is a symptom, not necessarily the root cause.&lt;/li&gt;
&lt;li&gt;Always determine whether the traffic is legitimate or malicious before scaling.&lt;/li&gt;
&lt;li&gt;Protect the system with rate limiting, load balancing, caching, and security controls.&lt;/li&gt;
&lt;li&gt;Investigate logs, metrics, databases, caches, deployments, and external dependencies after stability is restored.&lt;/li&gt;
&lt;li&gt;Every production incident is an opportunity to improve architecture, monitoring, and operational readiness.&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>systemdesign</category>
      <category>backend</category>
      <category>devops</category>
      <category>softwareengineering</category>
    </item>
  </channel>
</rss>
