On Tuesday 21 October 2026, at 12:18 in the afternoon, exams were about to start at several schools and colleges. Students sat down, opened their exam, and nothing happened. The start button did nothing, or the page never loaded. Three institutions were affected, up to 2,100 students in total, and for 19 minutes nobody at those institutions could start an exam.
The day before, the same thing had happened to one institution, for 13 minutes. A restart fixed it. On the Tuesday, the team restarted the servers twice. Nothing changed. The service came back only when they changed how the servers connected to the database.
Schoolyear is the platform those students use to take exams. Schools run exams on it, students start their exams through it, and supervisors open a dashboard to watch over them. Other assessment platforms connect to it too, sending requests on behalf of their own users. When Schoolyear stops answering, exams do not start.
The postmortem, published by Schoolyear itself, is short and unusually honest. It names two flaws, and both of them are the kind of flaw that looks sensible until the worst day arrives.
The burst
At the start of an exam, one integration sent a very large burst of requests. How large? Schoolyear does not give a number, but it says the burst was far larger than the number of students taking the exam would explain, and far beyond its fair-use limits. Exam size was not the problem: Schoolyear regularly runs exams with more students than were involved here.
A burst like that is exactly what rate limiting exists for. A rate limit is a rule that says one sender may make this many requests in this much time, and no more. Go over the limit and your extra requests are turned away, usually with a 429 response, which is the server saying "too many requests, try again later".
Schoolyear had a rate limit, and the rate limit worked. The extra requests were rejected. So why did the service fall over?
Because of what the rejection cost.
Flaw one: the bouncer that phoned the database
Think of a rate limiter as a bouncer at a door. The bouncer's job is to keep the room from getting overcrowded. People arrive, the bouncer counts, and once the room is full the bouncer turns people away at the door. Turning someone away should be the cheapest thing the bouncer does. No seat, no drink, no service. Just "no".
Schoolyear's bouncer did something extra before saying no. Before deciding whether to turn a request away, it checked the database.
That sounds harmless. Checking a limit usually means looking something up: how many requests has this sender made in the last minute? If that count lives in the database, every request, including every rejected request, costs a database lookup. The rejected requests still put load on the database. The protection cost as much as the thing it was protecting against.
Under normal traffic this does not matter. A few rejected requests, a few extra lookups, nobody notices. But a burst changes the arithmetic. When one sender fires off far more requests than the limit allows, nearly all of them are rejected, and nearly all of them still hit the database. The limiter was doing its job, and the database was paying for every rejection.
This is the first lesson, and it is worth stating plainly: a rate limiter that consults the database to reject a request is not protecting the database. It is just moving the load from the application to the database, one lookup at a time.
Flaw two: come back in one second
The second flaw was in the rejection itself. When Schoolyear turned a request away, it told the sender to retry after one second.
One second sounds reasonable. It is short, polite, and specific. The problem is what happens when thousands of rejected requests all get the same instruction. They all come back one second later, together, in step. The burst that was rejected at 12:18:00 returns at 12:18:01, is rejected again, and is told to come back at 12:18:02. The same crowd, knocking on the same door, once a second, until something gives.
The postmortem describes the loop in one sentence: retries added load, the load made requests slower, slower requests failed, and the failures produced more retries. The database eventually ran out of connections and the service stopped responding.
A fixed retry delay synchronises retries. Instead of every client retrying at a random moment, they all retry at the same moment, and the server gets the burst again, once a second, for as long as the clients keep trying.
Why the restarts did not work
On Monday, a restart fixed it. On Tuesday, two restarts did nothing. That difference is a clue.
Restarting a server clears whatever is stuck inside that server. It does not stop the requests arriving from outside. On Monday the burst had presumably passed by the time the servers came back, so the restart worked. On Tuesday the burst, and the retries it was generating, were still arriving. Each restarted server woke up to the same flood of rejected requests, each one costing a database lookup, and the database ran out of connections again.
Service returned at 12:37, after the servers were restarted with a different database connection, one that bypassed the component between the servers and the database that had become overloaded. The postmortem does not name that component, and this article will not guess. The point is the shape of the fix: when the path to the database is the thing that is choking, restarting the servers at the end of that path changes nothing. You have to change the path.
Exams already in progress were not affected, and no data was lost or exposed. The damage was bounded to exams that could not start. That is small comfort to 2,100 students sitting at a desk waiting for a page to load, but it matters: the failure was a failure to start, not a failure that corrupted anything.
What Schoolyear changed
All four fixes were live by 22 October:
- Servers now connect to the database directly, without the component that became overloaded.
- A single source can make fewer requests, so unusually large bursts are cut off sooner.
- The app backs off further after each retry, instead of retrying on a fixed schedule.
- Rate limiting no longer touches the database. The decision to reject a request is made without a lookup first, so excess requests are dropped before they reach the database.
Number 4 is the one that matters most, because it is the one that removes the flaw rather than shrinking it. A lower limit still lets a burst through, just a smaller one. Backoff still lets retries arrive, just spread out. But a limiter that decides without the database cannot be turned into a database load generator, however large the burst gets.
The code: a limiter that never touches the database
The fix is a token bucket kept in memory. A token bucket is a simple idea: each sender has a bucket holding a fixed number of tokens. Every request costs one token. Tokens refill at a steady rate. When the bucket is empty, the request is rejected, and the rejection costs nothing but a map lookup in memory.
In Go:
// Limiter decides in memory. It never touches the database,
// so rejecting a request costs a map lookup and nothing else.
type Limiter struct {
mu sync.Mutex
buckets map[string]*bucket
rate float64 // tokens refilled per second
burst int // maximum tokens in a bucket
}
type bucket struct {
tokens float64
last time.Time
}
func (l *Limiter) Allow(key string) bool {
l.mu.Lock()
defer l.mu.Unlock()
b, ok := l.buckets[key]
if !ok {
b = &bucket{tokens: float64(l.burst), last: time.Now()}
l.buckets[key] = b
}
now := time.Now()
b.tokens += l.rate * now.Sub(b.last).Seconds()
if b.tokens > float64(l.burst) {
b.tokens = float64(l.burst)
}
b.last = now
if b.tokens < 1 {
return false // rejected: no database call, no queue, no work
}
b.tokens--
return true
}
The same shape in Python, using a lock and a dictionary:
import threading
import time
class Limiter:
"""In-memory token bucket. Allow() never touches the database,
so a rejected request costs one dict lookup and nothing else."""
def __init__(self, rate, burst):
self.rate = rate # tokens refilled per second
self.burst = burst # maximum tokens in a bucket
self.buckets = {}
self.lock = threading.Lock()
def allow(self, key):
with self.lock:
now = time.monotonic()
tokens, last = self.buckets.get(key, (float(self.burst), now))
tokens = min(self.burst, tokens + self.rate * (now - last))
if tokens < 1:
self.buckets[key] = (tokens, now)
return False # rejected before any database work
self.buckets[key] = (tokens - 1, now)
return True
And on the client side, the second flaw has a matching fix. When a server says "retry after one second", do not take it literally for every retry. Back off further after each attempt, and add a random spread so clients do not all return at once:
import random
def backoff(attempt, base=1.0, cap=60.0):
"""Wait longer after each rejection, with jitter so retries
spread out instead of arriving in step."""
wait = min(cap, base * (2 ** attempt))
return wait * random.uniform(0.5, 1.5)
The server side of that fix is just as small: send a Retry-After that grows, or let the client back off on its own. Either way, the fixed one-second delay is the thing to remove.
The test that would have caught it
One test would have caught the first flaw before it ever reached production: flood the limiter with requests that will be rejected, and watch the database.
Send ten times the limit from one sender, then check how many database connections the flood used. If the answer is anything above zero, the limiter is not protecting the database. It is billing the database for every rejection. The test is cheap, and it measures exactly the property that failed: rejected requests must cost the database nothing.
Test the second flaw the same way. Send a burst, count the retries in the next ten seconds, and check whether they arrive in step or spread out. If they arrive in a single spike one second after the rejection, the retry delay is synchronising them.
What to take away
The rate limit worked. That is the part of this story that should stay with you. The limit was in place, it rejected the excess, and the service still went down twice, because the rejection itself was expensive and the retry instruction was too precise.
Protection that costs as much as the thing it protects is not protection. It is a queue with a nicer name. If you run a rate limiter, the question is not whether it rejects the excess. It is what a rejected request costs, and whether a thousand rejections in one second cost a thousand database lookups. If the answer is yes, the limiter is the load, and the next burst will prove it.
When did you last check what your rate limiter costs when it says no?
Sources: Schoolyear, "Incident Postmortem on 20-10-2026 and 21-10-2026", Schoolyear help centre. Incident windows 20 October 2026, 18:48 to 19:01 CEST, and 21 October 2026, 12:18 to 12:37 CEST. The postmortem does not name the component between the servers and the database, and does not give the size of the request burst, so this article does not guess at either. The Go and Python snippets show the shape of the fix, not Schoolyear source code.
Top comments (0)