You create a new Kubernetes cluster. You wait. Nothing crashes. Nothing errors in a useful way. The cluster just sits there, with core components pending, as if it is waiting for something that will never arrive.
That is how this one started, on October 17, 2026, for people creating new Linode Kubernetes Engine-Enterprise clusters, known as LKE-E. Existing clusters kept working. Ongoing workloads kept working. Scaling existing clusters kept working. Only new clusters were stuck.
That split is the first clue. When old things work and new things do not, the problem is usually not the whole system. It is in the path that only new things take.
What happened
Starting October 17, 2026, customers were unable to create new LKE-E clusters globally. The impact was limited to newly provisioned clusters, while existing clusters, ongoing customer workloads, and cluster scaling remained fully functional throughout the event.
Investigation found the trigger after Linode Virtual Private Cloud subnet IDs crossed 1,000,000, following growth on October 17, 2026.
A Virtual Private Cloud, or VPC, is a private network inside a cloud provider. A subnet is a slice of that network. Each subnet gets an ID, a label that other parts of the system use to find it.
Due to a software logic flaw, the Cloud Controller Manager parsed these 7-digit IDs into scientific notation instead of raw integers, rendering the IDs invalid and causing required core components to remain pending on newly created clusters.
The Cloud Controller Manager is the part of Kubernetes that talks to the cloud provider. It asks for load balancers, routes, and network resources, and it keeps those resources in step with what the cluster wants. If it cannot match a subnet ID, it cannot finish that work, so the components that depend on it stay pending.
Akamai deployed a fix to enforce strict integer formatting and patched existing custom resources to restore the affected clusters. Internal testing confirmed that newly provisioned test clusters were fully operational, and the impact was considered mitigated at around 19:49 UTC on October 20, 2026. A postmortem update followed at 15:49 UTC on October 21, 2026.
That is the whole incident, in one paragraph. The interesting part is not Linode. It is the pattern, because it shows up in ordinary code far more often than it should.
An ID is not a number you do math on
An ID looks like a number, but you should treat it as a label. You do not add two user IDs together. You do not average subnet IDs. You only need the label to survive every hop unchanged, from the database, through the API, through JSON, through your code, and back out again.
A float breaks that promise.
A float, short for floating point number, is how computers store numbers with fractions. It is good for measurements. It is bad for exact labels, because it does not store every integer exactly, and because formatting a float can change how it looks.
Take 1000000. As an integer, it is seven digits, nothing more. As a float, a formatter can turn it into scientific notation, something like 1e+06. That means 1 times 10 to the power of 6. It is the same value to a human doing math, but it is not the same string. A downstream system that expects digits will reject it, or look up the wrong thing, or fail to match anything at all.
That is why the boundary matters. 999999 has six digits. 1000000 has seven. A bug that only appears when an ID grows by one digit can sit quietly for years, then break everything on the day a counter crosses that line. The system did not get worse that day. The data got bigger, and a hidden assumption finally broke.
You have probably seen a version of this before. A phone number stored as a number loses its leading zero. A credit card number stored as a number loses precision. A timestamp stored as a float loses the last digits. The lesson is the same every time. If you do not do math on it, do not store it as a number you can do math on.
The guard, in Go
In Go, do not parse an ID through float64. Parse it as an integer, and fail loudly if it is not digits.
raw := "1000000"
id, err := strconv.ParseInt(raw, 10, 64)
if err != nil {
return err
}
_ = id
That code is small on purpose. ParseInt with base 10 and 64 bits will reject text that is not a plain integer. If the input is "1e+06", it fails, and you find out at the boundary, not three systems later when a cluster is pending.
JSON needs the same care. By default, Go's JSON decoder turns numbers into float64. If an ID arrives in JSON, tell the decoder to keep numbers as tokens instead.
dec := json.NewDecoder(r)
dec.UseNumber()
var v struct {
SubnetID json.Number `json:"subnet_id"`
}
if err := dec.Decode(&v); err != nil {
return err
}
if _, err := strconv.ParseInt(v.SubnetID.String(), 10, 64); err != nil {
return err
}
UseNumber does not fix the ID for you. It stops the decoder from turning it into a float before you get a chance to check it. The ParseInt check after it is the guard.
The guard, in Python
Python has the same trap, in a different shape. JSON numbers become int or float depending on the value, and a generic formatter can still change the form. The guard is to reject a float ID, and to validate the digit form before you use it.
def subnet_id(value):
if isinstance(value, bool):
raise ValueError("subnet id must be digits, not a boolean")
if isinstance(value, float):
raise ValueError("subnet id must not come through as a float")
text = str(value)
if not text.isdigit():
raise ValueError("subnet id must be digits only")
return int(text)
The boolean check looks odd until you remember that in Python, bool is a subclass of int, so True would otherwise pass as 1. That is exactly the kind of small detail that makes a boundary guard worth writing once and reusing.
If you want to keep the ID as a string all the way through, that is often the best choice. A string does not get reformatted by a number formatter, and it does not lose leading zeros. Store it as a string, pass it as a string, and only convert it when you have a real reason to, and a check to go with it.
The test that would have caught it
The fix here was strict integer formatting. The test that catches this class of bug is even simpler. Test at the boundary where the form changes.
Test with 999999. Test with 1000000. Test with 1000001.
If your code passes the first and fails the second, you have found the bug before your users do. If you only test with small IDs like 1, 2, and 42, you will never see it, because small IDs survive almost every bad conversion.
Add that test at the place where an ID enters your system, not just where it leaves. By the time a bad ID reaches the cloud controller, the database, or the downstream service, you have lost the context that would tell you where it changed form. Check it at the boundary, log the original form when it fails, and fail loudly. A pending cluster with no useful error is worse than a clear rejection, because it looks like slowness, not a bug.
There is a second lesson in this incident, and it is about how you read a failure. Existing clusters worked. New clusters did not. That tells you the core system was healthy, and the creation path was not. When you see that split, do not restart everything. Trace the path that only the failing case takes, and check every place where a value changes form along that path. In this case, that path was subnet creation, and the form change was integer to scientific notation.
What to check this week
Look for IDs in your own system that pass through a float, a generic number formatter, or a JSON parser that does not preserve the number form. That includes subnet IDs, user IDs, order IDs, and any other identifier that looks like a number but is really a label.
Ask three questions for each one. First, does anything do math on this ID? If not, why is it a number at all? Second, what happens when it crosses a power of ten, from 999 to 1000, or from 999999 to 1000000? Third, if the form changes, where do you find out, at the boundary, or days later when something is pending?
If you cannot answer the third question, that is the bug to fix first. A clear error at the boundary is a feature. A silent pending state is not.
Sources: Linode status summary for the October 20, 2026 LKE-E incident, as reproduced on Statusfield, https://statusfield.com/services/linode/incidents
Where in your own system does an ID pass through a float, a generic formatter, or a JSON parser that could change its form?
Top comments (0)