Still getting 504 errors in GKE even when everything looks fine?
We faced the same issue. Pods were healthy, APIs were responding, and there were no errors in logs.
The real problem wasn’t the application. It was a 30-second timeout.
We spent hours debugging this.
Everything looked normal:
- Pods were running fine
- APIs were responding
- No crashes or error logs
But still, requests kept failing.
We were seeing random 503 and 504 errors.
The Confusing Part
- Short requests worked perfectly
- Logs showed no issues
- System appeared healthy
But anything taking more than around 30 seconds failed every time.
What We Thought Initially
At first, we assumed:
- Application bug
- Database latency
- Resource limits
We checked everything. Nothing was wrong.
The Real Cause
After digging deeper, we found the actual issue:
GKE Load Balancer has a default timeout of 30 seconds.
This means:
- Your backend may still be processing
- But the load balancer stops waiting
Result: 504 Gateway Timeout.
The Fix: BackendConfig
We fixed it using BackendConfig.
It allows you to:
- Increase timeout
- Customize health checks
- Control load balancer behavior
What We Changed
apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
name: my-backend-config
spec:
timeoutSec: 60
This change increased the timeout from 30 seconds to 60 seconds.
After applying it, the issue was resolved.
Attach It to Your Service
annotations:
cloud.google.com/backend-config: '{"ports": {"8003":"my-backend-config"}}'
Apply the configuration and redeploy your service.
Important Note
If you don’t configure BackendConfig:
- Default timeout remains 30 seconds
- Long-running requests will fail
- You have limited control over behavior
Bonus Tip
If your application regularly takes longer to respond:
- Avoid increasing timeout too much
- Consider using asynchronous processing (queues or background jobs)
Final Thought
This was not a complex bug.
It was a default setting that we were not aware of.
Sometimes the real issue is not in your code, but in infrastructure defaults.
If you work with GKE, this can save you a lot of time.
Top comments (1)
If anyone is debugging 504 errors in GKE, check the load balancer timeout first.
This was the exact issue for us.