Everyone learns gradient descent early. It becomes the default model for how machine learning works.
Then you dig into decision trees and there is no gradient descent at all.
Trees learn by greedy split search. At each node, the algorithm tests thousands of thresholds, picks the one that most reduces impurity, and repeats.
This matters more than it sounds.
Because of the search is greedy, early splits carry massive weight. A weak top-level split where the tree first divides your data into every branch below it. You can't fix a bad root split by tuning later layers.
Working in ADAS, this introduces a major vulnerability. A small shifts in training data like a new weather condition, can produce an entirely different tree with a completely different decision structure.
Optimization method determines failure mode. Knowing how a model learns tells you exactly how it will break.
What’s a foundational assumption about ML optimization you held early on that you later had to unlearn?
Top comments (0)