I Have 15 Minutes to Save a Login System. Here's What I'd Do.
Scenario
Your authentication service normally handles 500 login requests per second. Suddenly, traffic spikes to 20,000 login requests per second. The CPU is stuck at 100%. Users can't log in. Customer support is flooded. Your manager asks, "Can you fix it?" You have 15 minutes.
What do you do?
Let's walk through this like we're the engineer on call.
Before Touching Anything, Understand the Problem
The biggest mistake during production incidents is assuming you already know the cause.
A login failure can happen because of dozens of reasons.
The CPU hitting 100% is only a symptom.
The real problem could be:
- A sudden marketing campaign
- A celebrity mentioning your app
- A bot attack
- Credential stuffing attack
- A DDoS attack
- Database slowdown
- Cache outage
- Recent deployment bug
- Third-party authentication provider failure
- Infrastructure misconfiguration
If you immediately restart the service without understanding what's happening, you might temporarily reduce CPU usage but make recovery even harder.
The first goal is not fixing the root cause.
The first goal is stopping the bleeding.
What Is Happening?
Let's compare the numbers.
| Metric | Normal State | Incident State |
|---|---|---|
| Traffic | 500 logins/sec | 20,000 logins/sec |
| CPU Usage | Healthy | 100% |
| User Impact | Users login normally | Users can't login |
That's a 40x traffic spike.
Most authentication services are designed for predictable traffic.
Receiving forty times the normal load overwhelms nearly every part of the system:
- CPU
- Memory
- Database
- Network
- Session storage
- Cache
- External identity providers
Even if only one component becomes overloaded, the entire login flow can fail.
Why Is This Dangerous?
When users cannot log in:
- Customers cannot access the application.
- Revenue may stop.
- Support tickets increase rapidly.
- Frustration spreads on social media.
- Business reputation suffers.
- Engineers are forced into emergency response mode.
For many companies, the login service is the front door.
If people cannot enter, nothing else matters.
Reasonable Assumptions
Since we only have the scenario, let's make a few reasonable assumptions:
- The application was working normally before.
- Infrastructure monitoring is available.
- Logs and metrics exist.
- The production team can scale infrastructure if necessary.
- The authentication service is stateless.
- The database stores user credentials.
If any of these assumptions are false, the response may differ.
The First 5 Minutes: Stabilize the System
Notice I didn't say "fix the problem."
At this stage, stability matters more than perfection.
1. Declare an Incident
The first action isn't technical.
It's operational.
Notify the team immediately.
Bring in:
- Backend engineers
- DevOps/SRE
- Database administrator (if needed)
- Security team (if an attack is suspected)
Production incidents are team sports.
2. Check Monitoring Dashboards
Look at:
- CPU
- Memory
- Network traffic
- Error rate
- Response time
- Request rate
- Database connections
- Cache hit ratio
You're trying to answer:
Where is the bottleneck?
3. Verify Whether the Traffic Is Legitimate
Ask questions like:
- Did marketing launch a campaign?
- Did a mobile app update just release?
- Did a popular influencer mention us?
- Is a partner integrating with our API?
Or...
Is someone attacking us?
This changes everything.
4. Protect the Login Service
If traffic is overwhelming the system, prevent it from getting worse.
Possible actions:
- Enable rate limiting.
- Block abusive IP addresses.
- Enable Web Application Firewall (WAF) rules.
- Activate bot protection.
- Reject excessive requests.
The goal is to preserve capacity for genuine users.
Think of it like crowd control at a stadium.
Instead of allowing everyone to rush the entrance at once, security lets people enter in an orderly way.
5. Scale the Authentication Service
If the traffic is legitimate and the application supports horizontal scaling:
Increase the number of authentication service instances.
More servers mean the incoming requests are shared across multiple machines instead of one overloaded server.
However...
Scaling isn't always enough.
If every new server still queries the same overloaded database, you've simply moved the bottleneck.
The Next 15 Minutes: Tactical Actions
Once the immediate pressure is under control, start investigating more deeply.
Check Error Logs
Look for patterns.
Examples:
- Database timeout
- Authentication timeout
- Connection pool exhausted
- Out of memory
- Thread exhaustion
- SSL failures
- Third-party authentication errors
Logs often reveal whether failures are occurring before, during, or after authentication.
Examine Database Health
Authentication usually depends on a database.
Check:
- CPU
- Active connections
- Slow queries
- Locks
- Replication lag
- Disk usage
If the database is overloaded, adding more application servers won't help.
Check the Cache
Many authentication systems cache:
- Session data
- User profiles
- Tokens
- Permissions
If the cache fails, every login request hits the database directly.
Imagine a library where the index cards disappear.
Now everyone has to search every shelf manually.
That's what happens when a cache goes down.
Check Recent Deployments
Ask:
"What changed today?"
Many production incidents begin immediately after:
- New release
- Configuration change
- Feature flag enabled
- Infrastructure update
- Database migration
If the timing matches, rolling back the deployment may be the fastest recovery option.
Investigate Third-Party Dependencies
Modern authentication often relies on external services.
Examples:
- OAuth providers
- Identity providers
- Multi-factor authentication services
- Email or SMS providers
If one of these is unavailable, login failures may not be your fault.
What If This Is a Security Attack?
A sudden 40x increase in login attempts could indicate:
- Credential stuffing
- Password spraying
- Bot attack
- DDoS attack
In that case:
- Notify the security team.
- Block malicious traffic.
- Enable CAPTCHA where appropriate.
- Tighten rate limits.
- Monitor suspicious IP ranges.
- Protect user accounts from brute-force attempts.
Treating an attack as normal traffic can make the situation much worse.
Common Technical Techniques That Help
Rate Limiting
Limits how many login requests a user or IP address can make within a certain time.
This prevents a few bad actors from consuming all available resources.
Load Balancing
A load balancer distributes incoming requests across multiple application servers.
Instead of one server handling everything, the work is shared.
Think of it like opening more checkout counters in a supermarket.
Auto Scaling
Cloud platforms can automatically create additional servers when CPU usage or traffic exceeds predefined thresholds.
This helps absorb sudden traffic spikes without manual intervention.
Caching
Store frequently accessed information in fast memory instead of repeatedly querying the database.
This reduces database load and improves response times.
Circuit Breakers
If a downstream service becomes slow or unavailable, a circuit breaker temporarily stops sending requests to it.
This prevents failures from cascading through the entire system.
Queueing Non-Critical Work
Some login-related tasks don't need to happen immediately.
For example:
- Audit logging
- Analytics events
- Welcome emails
- Notification updates
Move these to asynchronous queues so the login request can complete faster.
Root Cause Investigation
After users can log in again, begin the detailed investigation.
Review:
- Application logs
- Infrastructure metrics
- Distributed tracing
- Database performance
- Cache metrics
- Recent deployments
- Network changes
- Security alerts
- Third-party service status
- Capacity planning reports
Ask:
Why did this happen?
Then ask an even better question:
Why wasn't the system prepared for it?
That's where the biggest learning happens.
Long-Term Prevention
A mature engineering team doesn't stop after restoring service.
They improve the system so the same incident is less likely to happen again.
Capacity Planning
Estimate future traffic growth and ensure infrastructure can handle expected peaks.
Load Testing
Simulate realistic traffic before releasing changes.
Don't assume the system can handle high load. Prove it.
Auto Scaling Policies
Configure automatic scaling based on CPU usage, request rate, or latency instead of relying on manual intervention.
Better Observability
Invest in:
- Dashboards
- Alerts
- Logs
- Metrics
- Distributed tracing
Good observability helps engineers detect problems before customers report them.
Smarter Rate Limiting
Apply limits based on users, IP addresses, geographic regions, or API keys to protect the system without blocking legitimate traffic.
Database Optimization
Reduce unnecessary queries, add indexes where appropriate, optimize slow queries, and consider read replicas if read traffic becomes a bottleneck.
Disaster Recovery Drills
Practice incident response regularly.
Teams that rehearse outages respond faster and with fewer mistakes when real incidents occur.
Common Mistakes During Incidents
- Restarting services without understanding the problem.
- Scaling application servers while ignoring an overloaded database.
- Focusing only on CPU instead of end-to-end system health.
- Ignoring the possibility of a security attack.
- Making multiple production changes simultaneously, making it impossible to identify what helped or hurt.
- Failing to communicate with stakeholders and customers during the outage.
- Skipping the post-incident review after recovery.
Traffic Spike Detected
│
▼
Is it legitimate traffic?
│
┌─────┴─────┐
│ │
Yes No
│ │
▼ ▼
Scale System Block Malicious Traffic
│ │
└─────┬─────┘
│
▼
System Stabilized?
│
┌─────┴─────┐
│ │
No Yes
│ │
▼ ▼
Find Bottleneck Root Cause Analysis
│
▼
Implement Long-Term Fixes
Key Takeaways
- The first priority during an incident is stabilization, not perfection.
- A CPU spike is a symptom, not necessarily the root cause.
- Always determine whether the traffic is legitimate or malicious before scaling.
- Protect the system with rate limiting, load balancing, caching, and security controls.
- Investigate logs, metrics, databases, caches, deployments, and external dependencies after stability is restored.
- Every production incident is an opportunity to improve architecture, monitoring, and operational readiness.
Top comments (0)