As our series continues, Deep-Dive Backend Systems Roadmap. I created the simplest weather API backend, calling the other API to get weather data and saving it in Redis for faster retrieval and to reduce API calls to the other service.
But I didn't think this design would fall apart so quickly. It took just a few requests for everything to go wrong. I knew this problem existed, but I never imagined it would show up this early. I will explain the issue in detail below in this blog post.
Basic setup for this:
Create a basic api which will act as a dummy weather api and send back JSON. So that we don't hit the rate limit of the actual weather api for this testing purpose.
func handleWeatherJSON(w http.ResponseWriter, r *http.Request) {
// 1. Read the JSON file
file, err := os.Open("data.json")
defer file.Close()
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusOK)
// 2. Send file data in the response
_, err = io.Copy(w, file)
}
func main() {
http.HandleFunc("/api/weather", handleWeatherJSON)
err := http.ListenAndServe(":3000", nil)
if err != nil {
panic(err)
}
}
Now let's start building the actual system that serves users. For this example, I'm using a simple Weather API, but you can replace it with any third-party service, like an AI API.
The idea is simple: whenever we call the third-party API, store its response in the cache. That way, if the user asks for the same data again or simply reloads the page, we can return the cached response instead of making another API call.
func callTheAPI(w http.ResponseWriter, r *http.Request) {
city := r.URL.Query().Get("city")
var data any
// 1. Check Cache
if cachedBytes, err := rdb.Get(r.Context(), city).Bytes(); err == nil {
json.Unmarshal(cachedBytes, &data)
json.NewEncoder(w).Encode(Response{Data: data, Msg: "Cache HIT"})
return
}
// 2. Call Upstream API
resp, _ := http.Get("https://localhost:3000")
defer resp.Body.Close()
json.NewDecoder(resp.Body).Decode(&data)
// 3. Save to Cache
dataBytes, _ := json.Marshal(data)
rdb.Set(context.Background(), city, dataBytes, 10*time.Minute)
// 4. Send Response
json.NewEncoder(w).Encode(Response{Data: data, Msg: "Fetched from API"})
}
func main() {
// 1. Setup Redis
rdb = redis.NewClient(&redis.Options{
Addr: "localhost:6379",
Password: "",
DB: 0,
})
// 2. Handle API
http.HandleFunc("/weather", callTheAPI)
if err := http.ListenAndServe(":8080", nil); err != nil {
panic(err)
}
}
For large-scale load testing, I wrote test cases in K6 to simulate 500 concurrent requests hitting this API at the same time.
import http from 'k6/http';
import { check, sleep } from 'k6';
import { Counter } from 'k6/metrics';
// Custom metrics to track cache hits and misses directly in the CLI summary
const cacheHits = new Counter('cache_hits');
const cacheMisses = new Counter('cache_misses');
export const options = {
scenarios: {
// Scenario 1: Prime the cache with a single request at t = 0s
primer: {
executor: 'per-vu-iterations',
vus: 1,
iterations: 1,
startTime: '0s',
exec: 'primeCache',
},
// Scenario 2: Stampede blast starting right at the expiration window
// If TTL is 10s, start at 10.5s or 11s to guarantee the key has expired
stampede: {
executor: 'shared-iterations',
vus: 500, // 500 concurrent goroutines/users
iterations: 500, // Total requests to blast
maxDuration: '10s', // Window to complete them
startTime: '10.5s', // Wait for the 10s TTL to cleanly expire
exec: 'blastEndpoint',
},
},
};
const TARGET_URL = 'http://localhost:8080/weather?city=nagpur';
// Function for Scenario 1
export function primeCache() {
const res = http.get(TARGET_URL);
check(res, {
'primer status is 200': (r) => r.status === 200,
});
console.log(`[PRIMER] Cache primed at ${new Date().toISOString()}`);
}
// Function for Scenario 2
export function blastEndpoint() {
const res = http.get(TARGET_URL);
// Validate response status
check(res, {
'status is 200': (r) => r.status === 200,
});
// Track cache status from response headers
// Assumes your Go handler sends: w.Header().Set("X-Cache", "HIT" | "MISS")
const cacheHeader = res.headers['X-Cache'];
if (cacheHeader === 'HIT') {
cacheHits.add(1);
} else if (cacheHeader === 'MISS') {
cacheMisses.add(1);
}
}
And when I run this, I run into these issues. Most of the requests got missed by the cache.
Symptoms & Results:cache_hits.....................: 287
cache_misses...................: 213
And because of these 213 cache misses, all 213 requests ended up calling the external API. That created a huge number of unnecessary external requests and increased the API response time.
Why did this happen?
Because all 500 requests arrived almost at the same time. There might be just a few milliseconds of difference, but the first 213 requests checked the cache, got null, and immediately called the external API. Each one fetched the exact same data and tried to store it in Redis. Meanwhile, the remaining 287 requests hit the cache after the data was already there and got the response directly.
BUT, BUT, BUT...
Just think about it for a second.
213 API requests hitting another external API at the exact same time. You'll hit the rate limit in the blink of an eye on a real Weather API.
Or forget the Weather API. Imagine 213 identical database queries running at the same time. A simple database will slow down immediately, and a smaller one might even start struggling under that load.
And the worst part? All 213 requests are doing the exact same work just to fetch the exact same data.
This is what you call, bro, the
THUNDERING HERD problem
We have talked about this in more detail in another blog. You can check it out; right now will be talking about how to resolve it.
One of the best solutions for this is Single Flight.
Out of all 500 requests, only one request goes to the external API. The other 499 requests just wait on the server for that first request to finish.
Once that request gets the response, every waiting request uses the same data instead of calling the external API again. We also store that response in the cache, so any future request can get it directly from Redis without hitting the external API.
Just write a simple single-flight program which will hold the requests until the data comes in, or use any single-flight library.
func callTheAPI(w http.ResponseWriter, r *http.Request) {
city := r.URL.Query().Get("city")
var data any
// 1. Check Cache
if cachedBytes, err := rdb.Get(r.Context(), city).Bytes(); err == nil {
json.Unmarshal(cachedBytes, &data)
json.NewEncoder(w).Encode(Response{Data: data, Msg: "Cache HIT"})
return
}
// 2. Use SingleFlight to prevent concurrent duplicate API calls
result, _, _ := sfGroup.Do(city, func() (interface{}, error) {
var innerData any
// Call Upstream API
resp, _ := http.Get("https://github.com")
defer resp.Body.Close()
json.NewDecoder(resp.Body).Decode(&innerData)
// Save to Cache
dataBytes, _ := json.Marshal(innerData)
rdb.Set(context.Background(), city, dataBytes, 10*time.Minute)
return innerData, nil
})
// 3. Send Response
json.NewEncoder(w).Encode(Response{Data: result, Msg: "Fetched from API / SingleFlight"})
}
ONLY FOR GO USERS
One important thing: if you're saving data to Redis inside a Go goroutine, use a different background context, not the request context. The request context gets cancelled as soon as the request ends, so the Redis write can fail. A background context avoids that problem.
Whenever you're caching data in Redis, always think about the Thundering Herd problem. Use Single Flight so only one request fetches the data, while the rest wait and reuse the same response instead of hitting the external API again.
Top comments (0)