DEV Community: Fit Happens ML

Every tree and boosting formula for ML interviews. Part 3: theory

Fit Happens ML — Thu, 09 Jul 2026 07:00:20 +0000

July 2026. Math doesn't rot, but library details do - XGBoost version notes reflect 2.x-3.1.

The whole series:

Part 1 - the Interview tier: trees, forests, and the ten core formulas
Part 2 - the Practice tier: the knobs you actually turn
Part 3 - the Theory tier: full derivations and the cheat sheets (you are here)

This is the theory tier: the full derivations behind everything stated in part 1 and part 2, plus the cheat-sheet appendices. If an interviewer ever answers your answer with "why?", this is the part that has your back. Cross-references like I.7 point to part 1, P.3 to part 2.

TIER 3 - [T] THEORY

Full derivations. What lets you answer any follow-up question.

[T.1] The regularized XGBoost objective

Obj = i = 1 \sum n L (y_{i}, y^_{i}) + k = 1 \sum N Ω (f_{k}), Ω (f) = γ T + \frac{1}{2} λ j = 1 \sum T w_{j 2} + α j = 1 \sum T ∣ w_{j} ∣

The key difference from GBM. In classical GBM regularization is heuristic and external (cap the depth, apply shrinkage). Here the tree's complexity is an explicit term of the objective, so the algorithm itself decides, arithmetically, whether a split pays for itself.

Three components:

$γ T$ - the price of leaf count => becomes the " $- γ$ " in Gain I.8;
$\frac{1}{2} λ \sum w_{j 2}$ - ridge on leaf weights => the $λ$ in the denominator of $w^{*}$ I.7;
$α \sum ∣ w_{j} ∣$ - lasso on leaf weights => soft-thresholding T.4.

Why this is ridge/lasso "for real". In a linear model the parameters are the coefficients $β$ on features; in a tree the parameters are the leaf weights $w_{j}$ . We penalize exactly those, in exactly the same sense. Different architectures, one idea.

[T.2] The second-order Taylor expansion

Substitute the additive step:

Obj^{(t)} = i = 1 \sum n L (y_{i}, y^_{i (t - 1)} + f_{t} (x_{i})) + Ω (f_{t}) + const

Expand $L$ in the small increment $f_{t} (x_{i})$ around $y^_{i (t - 1)}$ :

L (y_{i}, y^_{i (t - 1)} + f_{t} (x_{i})) \approx L (y_{i}, y^_{i (t - 1)}) + g_{i} f_{t} (x_{i}) + \frac{1}{2} h_{i} f_{t} (x_{i})^{2}

where

g_{i} = \frac{\partial L}{\partial y ^ _{i}} ∣ ∣ y^^{(t - 1)}, h_{i} = \frac{\partial ^{2} L}{\partial y ^ _{i 2}} ∣ ∣ y^^{(t - 1)}

Dimensions. $g, h \in R^{n}$ (vectors, one number per sample). For multiclass - $n \times K$ .

Contrast with GBM. Classical boosting uses only $g$ (gradient descent in function space). XGBoost adds $h$ - the curvature => that's Newton's method, hence "Newton boosting".

What the second order buys.

An analytic optimum for the leaf weight (a closed formula instead of line search).
A split criterion consistent with the loss.
A more accurate step (curvature is taken into account).

Limitations. Requires $h_{i} > 0$ (otherwise the denominator breaks). For non-convex custom losses the hessian gets clipped. For MAE/quantile, where $h$ is constant, the Newton step degenerates => adaptive trees P.1.

[T.3] Regrouping by leaves and deriving the optimal leaf weight

Step 1. The constant $L (y_{i}, y^_{i (t - 1)})$ doesn't depend on $f_{t}$ => drop it:

Obj^{(t)} \approx i = 1 \sum n [g_{i} f_{t} (x_{i}) + \frac{1}{2} h_{i} f_{t} (x_{i})^{2}] + γ T + \frac{1}{2} λ j = 1 \sum T w_{j 2}

Step 2 (the key trick). A tree predicts a constant within a leaf: if $i \in I_{j}$ , then $f_{t} (x_{i}) = w_{j}$ . So the sum over samples can be rewritten as a sum over leaves:

Obj^{(t)} \approx j = 1 \sum T ⎣ ⎡ (\sum_{i \in I_{j}} g_{i} G_{j}) w_{j} + \frac{1}{2} (H_{j} \sum i \in I_{j} h_{i} + λ) w_{j 2} ⎦ ⎤ + γ T

This is the moment that makes the problem solvable: the leaves become independent, each optimized separately.

Step 3. Inside a leaf - an ordinary parabola:

ϕ (w_{j}) = G_{j} w_{j} + \frac{1}{2} (H_{j} + λ) w_{j 2}

ϕ^{'} (w_{j}) = G_{j} + (H_{j} + λ) w_{j} = 0 ⟹ w_{j *} = - \frac{G _{j}}{H _{j} + λ}

Convexity check. $ϕ^{''} (w_{j}) = H_{j} + λ > 0$ whenever $h_{i} \geq 0$ and $λ > 0$ => it's a minimum, not a maximum.

[T.4] Soft-thresholding: how the L1 case is solved

With the $α ∣ w_{j} ∣$ term, the leaf's objective is:

ϕ (w_{j}) = G_{j} w_{j} + \frac{1}{2} (H_{j} + λ) w_{j 2} + α ∣ w_{j} ∣

The absolute value is non-differentiable at zero => solve via the subgradient. The result is the classic soft-thresholding, as in lasso:

w_{j *} = - \frac{T α ( G _{j} )}{H _{j} + λ}, T α (G) = sign (G) \cdot max (∣ G ∣ - α, 0)

Spelled out:

Condition	$T_{α} (G)$
$G > α$	$G - α$
$G < - α$	$G + α$
$∣ G ∣ \leq α$	$0$

Meaning. If the total gradient in a leaf is no larger than $α$ in absolute value, the leaf's weight is zeroed out - the leaf predicts exactly zero and contributes nothing.

Comparing $λ$ and $α$ :

	L2 ( $λ$ )	L1 ( $α$ )
Where it sits	denominator	numerator (a threshold)
Effect	proportional shrinkage	shift toward zero, exact zeroing
Sparsity	no	yes (zero leaves)
Default	1	0

Why $α$ is less useful in practice than in linear models. In linear regression, L1 zeroes coefficients on features - that's feature selection. In a tree the features are already selected by the structure (the splits); zeroing a leaf's weight doesn't remove a feature from the model, it merely mutes one leaf. Hence the practice: nonzero $λ$ , $α$ usually zero.

Structure score with L1: replace $G^{2}$ with $[T_{α} (G)]^{2}$ .

Where else it shows up. Soft-thresholding is the core of ISTA / proximal gradient for lasso, and the solution of $min_{w} \frac{1}{2} (w - a)^{2} + α ∣ w ∣$ in any context.

[T.5] The structure score

Substitute $w_{j *}$ back into the objective (without L1, for cleanliness). The parabola's minimum is $ϕ (w^{*}) = - \frac{G _{j 2}}{2 ( H _{j} + λ )}$ , hence:

Obj^{*} = - \frac{1}{2} j = 1 \sum T \frac{G _{j 2}}{H _{j} + λ} + γ T

What this is. A quality measure of a fixed tree structure $q$ (which splits, not which weights). The larger $\sum_{j} \frac{G _{j 2}}{H _{j} + λ}$ , the better (note the leading minus).

The philosophical moment. This is an impurity derived from the loss function, not postulated. Gini and entropy are heuristics; $\frac{G ^{2}}{H + λ}$ is a consequence. That is why XGBoost doesn't use Gini.

[T.6] Deriving Gain from the structure score

Consider a leaf $I$ we're thinking of splitting into $I_{L}$ and $I_{R}$ .

Before the split, its contribution to $Obj^{*}$ : $- \frac{1}{2} \frac{G ^{2}}{H + λ} + γ \cdot 1$ (one leaf).

After the split: $- \frac{1}{2} (\frac{G _{L 2}}{H _{L} + λ} + \frac{G _{R 2}}{H _{R} + λ}) + γ \cdot 2$ (two leaves).

The gain = (before) − (after), i.e. how much the objective decreased:

Gain = \frac{1}{2} [\frac{G _{L 2}}{H _{L} + λ} + \frac{G _{R 2}}{H _{R} + λ} - \frac{G ^{2}}{H + λ}] - γ

Notice: $γ \cdot 2 - γ \cdot 1 = γ$ - that's where the " $- γ$ " comes from. A global penalty $γ T$ produced a local rule.

Rule. Split when $Gain > 0$ .

[T.7] Equivalence to weighted least squares: Newton boosting ≈ IRLS

Take the post-Taylor objective and complete the square:

i \sum [g_{i} f (x_{i}) + \frac{1}{2} h_{i} f (x_{i})^{2}] = i \sum \frac{h _{i}}{2} [f (x_{i})^{2} + \frac{2 g _{i}}{h _{i}} f (x_{i})] = i \sum \frac{h _{i}}{2} (f (x_{i}) + \frac{g _{i}}{h _{i}})^{2} - const i \sum \frac{g _{i 2}}{2 h _{i}}

Altogether:

Obj^{(t)} \equiv i = 1 \sum n \frac{1}{2} h_{i} (f_{t} (x_{i}) - (- \frac{g _{i}}{h _{i}}))^{2} + const

What this means. Every boosting iteration is a weighted-least-squares fit of a tree to the targets $- g_{i} / h_{i}$ with weights $h_{i}$ .

That is literally IRLS (iteratively reweighted least squares), except the base learner is a tree instead of a linear model.

The parallel with logistic regression:

	Logistic regression	XGBoost
Method	Newton-Raphson / Fisher scoring / IRLS	Newton boosting
Gradient	$X^{⊤} (p - y)$ , size $p \times 1$	$g_{i} = p_{i} - y_{i}$ , a vector $n \times 1$
Curvature	$W = diag (p_{i} (1 - p_{i}))$ , $n \times n$	$h_{i} = p_{i} (1 - p_{i})$ , a vector $n \times 1$
Hessian	$X^{⊤} W X$ , size $p \times p$	$H_{j} = \sum h_{i}$ (a scalar per leaf)
Step	solves weighted LS analytically	approximates weighted LS with a tree
Regularization	ridge on $β$	ridge on $w_{j}$

Three consequences you should be able to say out loud.

Why the quantiles are weighted by $h_{i}$ P.12 - because the problem is weighted LS with weights $h_{i}$ .
Why $w^{*} = - G / (H + λ)$ is a weighted average of the individual Newton steps $- g_{i} / h_{i}$ with weights $h_{i}$ :

w_{j *} = \frac{\sum _{i \in I_{j}} h _{i} ( - \frac{g _{i}}{h _{i}} )}{\sum _{i \in I_{j}} h _{i} + λ}

Why $h = p (1 - p)$ in both - one and the same curvature of the logistic likelihood.

Note [T]. This is the strongest card in the whole deck. It shows that logistic regression and boosting are connected in your head.

[T.8] Initializing F0: what the "zeroth tree" predicts

F_{0} = ar g c min i = 1 \sum n L (y_{i}, c)

The first thing is not a tree - it's a constant (a global intercept; base_score in XGBoost).

Loss	$F_{0}$	Comment
MSE	the mean $y ˉ$	the "it's the mean" intuition is true exactly here
Log-loss	the logit of the base rate $lo g \frac{y ˉ}{1 - y ˉ}$	log-odds of the positive share
MAE	the median of $y$
Pinball ( $τ$ )	the $τ$ -quantile of $y$
Poisson (log link)	$lo g y ˉ$

Critical. $F_{0}$ lives in margin space (before the inverse link). For classification it's log-odds, not a probability.

The first real tree ( $f_{1}$ ) is trained on gradients relative to $F_{0}$ :

MSE: $g_{i} = F_{0} - y_{i} \Rightarrow$ the tree fits residuals from the mean. This is where "boosting fits residuals" is exact.
Log-loss: $g_{i} = p_{0} - y_{i}$ , where $p_{0} = σ (F_{0})$ is the base rate.

A historical XGBoost trap. Before version 2.0, base_score was a constant 0.5 (even for regression, where that's meaningless). And 0.5 is specified after the inverse link: for logistic tasks it's a probability that internally becomes $lo g \frac{0.5}{0.5} = 0$ . Since 2.0 the intercept is estimated automatically from the target; since 3.1 it's vector-valued for multi-output models.

base_margin. A vector of per-sample starting margins (overrides base_score). Used for stacking: feed another model's raw (pre-link!) predictions so the booster learns a correction on top.

Note [T]. With enough rounds, the value of $F_{0}$ barely matters - the model re-learns it. It matters with few trees, aggressive early stopping, and for calibration in early iterations. For a custom loss the auto-estimation doesn't work - set it by hand.

[T.9] AdaBoost: the exponential loss and the logit connection

AdaBoost (Freund & Schapire, 1997) was the first boosting algorithm that worked in the wild, and it made an entrance: Breiman, not a man given to hype, called it "the best off-the-shelf classifier in the world" (the quote survives via ESL). It also earned Freund and Schapire a Gödel Prize. Worth knowing not just as history: interviewers love asking how it relates to gradient boosting, and the honest answer (it's the same forward-stagewise frame under a specific loss) is below.

The algorithm (ESL 10.1), step by step.

Initialize weights $w_{i} = 1/ n$ .
For $m = 1.. M$ :
- fit a classifier $G_{m} (x) \in {- 1, + 1}$ under weights $w_{i}$ ;
- compute the weighted error

err m = \frac{\sum i w _{i} I ( y _{i} \neq = G _{m} ( x _{i} ))}{\sum _{i} w _{i}}

- compute the classifier's weight

α_{m} = lo g \frac{1 - err _{m}}{err _{m}}

- update the sample weights

w_{i} \leftarrow w_{i} \cdot exp (α_{m} I (y_{i} \neq = G_{m} (x_{i})))

Output: $G (x) = sign (\sum_{m} α_{m} G_{m} (x))$ .

Where $α_{m}$ comes from. AdaBoost is forward stagewise additive modeling with the exponential loss $L (y, f) = e^{- y f}$ , $y \in {- 1, + 1}$ . Minimizing over the coefficient gives $β_{m} = \frac{1}{2} lo g \frac{1 - err _{m}}{err _{m}}$ ; ESL's notation takes $α_{m} = 2 β_{m}$ , which is why the $\frac{1}{2}$ is absent.

The most beautiful fact - the population minimizer of the exponential loss:

f^{*} (x) = ar g f min E [e^{- y f (x)} ∣ ∣ x] = \frac{1}{2} lo g \frac{P ( y = 1 ∣ x )}{P ( y = - 1 ∣ x )}

So AdaBoost estimates half the log-odds! Hence the inverse transform $P (y = 1∣ x) = \frac{1}{1 + e ^{- 2 f (x)}}$ - a sigmoid with a factor of two.

Where else it shows up. This is the same quantity that logistic regression predicts (log-odds), the same thing that lives in a booster's margin. The exponential and logistic losses share the same population minimizer up to a constant factor.

The difference between the losses. The exponential loss grows faster than the logistic one at large negative margins => AdaBoost is more sensitive to outliers and label noise. That is the practical reason gradient boosting on log-loss wins in production.

Note [T]. Being able to say "AdaBoost = forward stagewise + exponential loss" and "its population minimizer is half the log-odds" is very strong. Most people only know the re-weighting procedure.

[T.10] Multiclass: softmax, gradients, the diagonal hessian

p_{ik} = \frac{e ^{z_{ik}}}{\sum _{c = 1 K} e ^{z_{i c}}}, L_{i} = - k = 1 \sum K y_{ik} lo g p_{ik}

Dimensions. $z_{i} \in R^{K}$ (per-class margins); $p_{i} \in R^{K}$ ; $y_{i}$ - one-hot $\in {0, 1}^{K}$ .

Gradient:

g_{ik} = p_{ik} - y_{ik}

The full hessian (a $K \times K$ matrix per sample):

\frac{\partial ^{2} L _{i}}{\partial z _{ik} \partial z _{i c}} = p_{ik} (δ_{k c} - p_{i c})

What XGBoost does. Uses the diagonal approximation, ignoring off-diagonal elements:

h_{ik} \approx 2 p_{ik} (1 - p_{ik})

(the factor 2 is an implementation detail insuring against oversized steps).

Cost. Each iteration builds $K$ trees - one per class. 100 rounds and 10 classes => 1000 trees. Cost grows linearly in the number of classes.

The alternative (XGBoost 3.1+). Vector leaf - a single tree whose leaf holds a vector of weights $w_{j} \in R^{K}$ . Lets the structure capture correlations between outputs. Still work in progress.

Note [T]. Knowing that the hessian is diagonally approximated is a rare detail and an excellent marker.

[T.11] SHAP: Shapley values

ϕ_{i} = S \subseteq N ∖ {i} \sum \frac{∣ S ∣ ! ( ∣ N ∣ - ∣ S ∣ - 1 ) !}{∣ N ∣ !} [f (S \cup {i}) - f (S)]

Symbols. $N$ - the set of all features ( $∣ N ∣ = d$ ); $S$ - a subset excluding feature $i$ ; $f (S)$ - the model's prediction with only the features in $S$ "known"; $ϕ_{i}$ - feature $i$ 's contribution.

The meaning of the multiplier. Averaging over all possible orders of adding features: $\frac{∣ S ∣ ! ( d - ∣ S ∣ - 1 )!}{d !}$ is the share of permutations in which feature $i$ arrives immediately after the set $S$ .

The additivity property (the main one):

f (x) = ϕ_{0} + i = 1 \sum d ϕ_{i}

where $ϕ_{0} = E [f (X)]$ is the base value. The prediction decomposes exactly into feature contributions.

The Shapley axioms (the unique solution satisfying them): efficiency (the additivity above), symmetry, dummy (a null feature => zero contribution), linearity.

Complexity. Naively $O (2^{d})$ . TreeSHAP exploits the tree structure and computes exact Shapley values in polynomial time. Available in XGBoost via pred_contribs=True.

The critical limitation. SHAP explains the model, not the world. It shows what the model looks at, not the causal structure of the data. If the model learned a proxy or a confounder, SHAP will faithfully show you that proxy. Causality does not follow from SHAP.

Where else it shows up. Cooperative game theory (Shapley's original 1953 setting): dividing a coalition's payoff among players.

[T.12] Functional gradient descent - the general frame

Generalizing I.6, boosting is descent in a space of functions:

F_{t} = F_{t - 1} - η \nabla_{F} L (F) ∣ ∣ F = F t - 1

where the "gradient" is the vector $(- g_{1}, ..., - g_{n})$ of negative-gradient values at the training points. Since we can't move in an arbitrary direction of an infinite-dimensional space, we project onto the space of trees:

f_{t} = ar g f \in F min i = 1 \sum n (- g_{i} - f (x_{i}))^{2}

In other words: the tree that best approximates the negative gradient. That's classical GBM.

XGBoost does the same, but in the metric given by the hessian:

f_{t} = ar g f \in F min i = 1 \sum n \frac{h _{i}}{2} (- \frac{g _{i}}{h _{i}} - f (x_{i}))^{2} + Ω (f)

The projection is now curvature-weighted - which is precisely the passage from gradient descent to Newton's method T.7.

The hierarchy of generalizations (worth drawing in your head):

forward stagewise additive modeling
├── exponential loss                     => AdaBoost   **T.9**
├── any differentiable L, 1st order      => GBM        **I.6**
└── any twice-diff. L, 2nd order + Ω     => XGBoost    **T.2-T.6**

Appendix A. The cross-connections - "where else does this show up"

Quantity / idea	In trees	Elsewhere
$p (1 - p)$	hessian of log-loss; Gini (binary, times 2)	Bernoulli variance; $W$ in $(X^{⊤} W X)^{- 1}$ of logistic regression; the IRLS weight
$- G / (H + λ)$	XGBoost leaf weight	the ridge solution $(X^{⊤} X + λ I)^{- 1} X^{⊤} y$
Soft-thresholding	L1 on leaf weights	lasso; ISTA / proximal gradient
Weighted least squares	every XGBoost iteration	IRLS in GLMs; Fisher scoring
Log-odds	the boosting margin; $F_{0}$ for log-loss	the linear predictor of logistic regression; population minimizer of the exponential loss (times $\frac{1}{2}$ )
$ρ σ^{2} + \frac{1 - ρ}{N} σ^{2}$	why RF decorrelates	risk of a portfolio of correlated assets
Bias-variance	bagging vs boosting	regularization in any model
"Importance ≠ effect"	feature importance, SHAP	a regression coefficient on observational data ≠ a causal effect
Base-rate shift	`scale_pos_weight`, undersampling	negative sampling breaks calibration; the intercept moves, not the slopes
Fisher information	$H_{j}$ as local curvature	$SE (β^) = diag (X^{⊤} W X)^{- 1}$

Appendix B. What gets centered - the quick card

Computing	Center?
Impurity, Gain, $G_{j}$ , $H_{j}$	no
Features before a tree	no (invariance to monotone transformations)
$X^{⊤} W X$ (Fisher information, logistic regression)	no (curvature, not covariance; the intercept centers implicitly)
$S_{x y} = \sum (x_{i} - x ˉ) (y_{i} - y ˉ)$ (CUPED, OLS)	yes, mandatory (it's a covariance)
Features before a regularized linear model	yes (the penalty depends on the scale of $β$ )

Mnemonic: centering belongs where a covariance of features is computed. Not where a loss curvature or sums of derivatives are.

Appendix C. The dimension checklist (self-test)

Expression	Dimension	Check
$X$	$n \times d$	no column of ones for a tree
$g$ , $h$	$n \times 1$	one number per sample
$G_{j}$ , $H_{j}$	scalars	sums over the leaf's samples
$w$	$T \times 1$	one number per leaf
$q (x)$	$R^{d} \to {1.. T}$	the tree structure
Gain	scalar	compared against zero
Multiclass hessian (full)	$K \times K$ per sample	XGBoost takes the diagonal
$W$ in logistic regression	$n \times n$ diagonal	don't confuse with $H_{j}$ !
$X^{⊤} W X$	$(d + 1) \times (d + 1)$	with the intercept
$ϕ$ (SHAP)	$d \times 1$ per sample	$f (x) = ϕ_{0} + \sum ϕ_{i}$

Appendix D. The 60-second checklist - what to say

A tree - low bias, high variance; greedy impurity-driven splitting; globally optimal trees are NP-hard.
Bagging/RF attacks variance: $Var = ρ σ^{2} + \frac{1 - ρ}{N} σ^{2}$ ; max_features lowers $ρ$ ; OOB comes for free.
Boosting attacks bias: an additive model, trees fit the negative gradient; it overfits => early stopping.
XGBoost: regularization inside the objective, second-order Taylor (Newton boosting), $w^{*} = - G / (H + λ)$ , Gain with $- γ$ , a criterion derived from the loss rather than Gini.
The connection: $h = p (1 - p)$ = the Fisher-information weight; every iteration = weighted least squares = IRLS with trees.
The limits: no extrapolation; probabilities aren't calibrated out of the box; importance ≠ causal effect.

That's the whole file. If it saved you a tab explosion the night before an interview, a ❤️ (or a unicorn, I don't judge) is the one signal that tells me these are worth making. And if I got something wrong, say so in the comments: a reference is only useful if it's right.

If write-ups like this are your thing, I announce the best ones on my Telegram channel, FitHappensML, along with discounts and free tiers from various providers whenever I spot them. No Telegram? I'm on X and Threads too. And if you're in ML/DS or building apps yourself, I follow back.

Every tree and boosting formula for ML interviews. Part 2: practice

Fit Happens ML — Thu, 09 Jul 2026 06:57:02 +0000

July 2026. Math doesn't rot, but library details do - XGBoost version notes reflect 2.x-3.1.

The whole series:

Part 1 - the Interview tier: trees, forests, and the ten core formulas
Part 2 - the Practice tier: the knobs you actually turn (you are here)
Part 3 - the Theory tier: full derivations and the cheat sheets

This is the practice tier of the reference: the parameters you actually turn when training tree ensembles, and the places where the intuitive reading of a knob is simply wrong. Notation and the ten core formulas live in part 1; cross-references like I.7 point there, T.7 points to part 3.

TIER 2 - [P] PRACTICE

What you actually turn, and where people actually get burned.

[P.1] The leaf value: CART, GBM (line search), XGBoost

Three different answers to "what's inside a leaf".

CART (a plain tree). Computed "automatically":

w_{j} = y ˉ_{I_{j}} (regression), w_{j} = ar g k max p_{k} (classification)

GBM (Friedman) - line search. The tree is grown by the ordinary CART criterion on pseudo-residuals, and then the leaf values are re-fitted by one-dimensional optimization inside each leaf:

γ_{j t} = ar g γ min i \in I_{j} \sum L (y_{i}, F_{t - 1} (x_{i}) + γ)

XGBoost - a closed formula. $w_{j *} = - G_{j} / (H_{j} + λ)$ , see I.7. This is the analytic solution of the same problem after the quadratic approximation.

Note [P]. Three stages of evolution: "the mean" => "one-dimensional search" => "a closed formula from Taylor". A good answer to "how does XGBoost differ from GBM".

Trap. For reg:absoluteerror (MAE) and quantile regression the hessian is constant, the Newton step degenerates => XGBoost uses adaptive trees: it builds the structure, then recomputes the leaf values as the median (or the quantile). In other words, it falls back to line-search logic.

[P.2] Learning rate (shrinkage)

y^^{(t)} = y^^{(t - 1)} + η f_{t} (x), η \in (0, 1]

Role. Shrinking each tree's contribution leaves "room" for the following ones. A form of regularization.

The trade-off. $η$ and the number of trees $N$ trade against each other: roughly $η \cdot N \approx const$ for comparable fit. Smaller $η$ + larger $N$ => better generalization, longer training.

Practice. $η \in [0.01, 0.1]$ ; the number of rounds is decided by early stopping, not by hand.

Note [P]. The standard recipe: fix $η = 0.05$ , tune everything else, then at the end cut $η$ by 2-4x and raise the round cap proportionally.

[P.3] `min_child_weight` is NOT a sample count

a split is rejected if H_{child} = i \in I_{child} \sum h_{i} < min child weight

The $g$ / $h$ table by objective (memorize):

Objective	$g_{i}$	$h_{i}$	What `min_child_weight` means
`reg:squarederror`, $L = \frac{1}{2} (y - y^)^{2}$	$y^_{i} - y_{i}$	$1$	number of samples in the leaf
`binary:logistic`	$p_{i} - y_{i}$	$p_{i} (1 - p_{i})$	total uncertainty
`count:poisson` (log link)	$e^{y^_{i}} - y_{i}$	$e^{y^_{i}}$	sum of expected intensities
`reg:absoluteerror`	$sign (y^_{i} - y_{i})$	$1$	number of samples

The intuition-breaking consequence. In binary classification, a leaf of ten confident samples ( $p \approx 0$ or $1$ , $h \approx 0$ ) has a tiny $H$ and will be rejected. A leaf of three borderline samples ( $p \approx 0.5$ , $h = 0.25$ ) gives $H = 0.75$ and may pass. The parameter doesn't say "few samples" - it says "not enough reliable information".

Scaling. $H$ grows with data size => the default of 1 restricts almost nothing at a million rows. Tune it with $n$ in mind.

Note [P]. A top-3 XGBoost trap question. Plenty of people confidently answer "minimum number of samples in a leaf" - true only for MSE.

[P.4] `max_delta_step`

∣ w_{j} ∣ \leq max delta step

Why. Under extreme imbalance in logistic tasks $h_{i} = p (1 - p) \to 0$ , the denominator $H + λ$ is tiny => the leaf weight blows up => unstable updates. A hard cap (typically 1-10) makes the step conservative.

Note [P]. The standard recommendation for heavily imbalanced binary classification, together with scale_pos_weight P.10. Default 0 = no cap.

[P.5] Cost-complexity pruning and the number of leaves

Classic CART (post-pruning):

R_{α} (T) = R (T) + α ∣ T ~ ∣

Symbols. $R (T)$ - the tree's training error; $∣ T ~ ∣$ - the number of leaves; $α \geq 0$ - the price of one leaf. As $α$ grows, the optimal subtree shrinks (a nested sequence of subtrees exists).

XGBoost - the same idea, but inside the objective:

Ω (f) = γ T + \frac{1}{2} λ j = 1 \sum T w_{j 2} + α_{L 1} j = 1 \sum T ∣ w_{j} ∣

Here $γ$ plays the role of CART's $α$ .

How $T$ relates to splits. A binary split turns one leaf into two: $Δ T = + 1$ . Hence the local price of a split $= γ$ I.8.

Step by step: how XGBoost actually prunes.

Grow the tree to max_depth (depthwise).
Walk bottom-up and prune nodes with $Gain < 0$ .

Why post-pruning rather than greedy stopping. A split can be bad on its own ( $Gain < 0$ ) yet open the path to an excellent split below it. Greedy stopping would miss that.

Note [P]. max_depth is a ceiling, not a target depth. Trees come out asymmetric: one branch deep, another cut early. Late trees in a boosting run are usually shallower (gradients are smaller => Gain hits zero sooner).

[P.6] `max_features` / `mtry` - the heuristics

classification: m \approx d, regression: m \approx \frac{d}{3}

Symbols. $d$ - total features, $m$ - features considered at each split.

Where from. Empirical recommendations (Breiman, ESL). This is the $ρ$ lever from I.3: smaller $m$ => more diverse trees => lower correlation => lower variance floor; but each tree is weaker (higher bias). The classic trade-off.

The XGBoost analogue - three levels that multiply:

m_{node} \approx d \cdot colsample bytree \cdot colsample bylevel \cdot colsample bynode

With all three at 0.5, a node sees roughly $0.125 d$ features.

Note [P]. Tying max_features to formula I.3 is a strong move. Not "we add randomness" but "we lower $ρ$ to push down the variance floor".

[P.7] Bootstrap size and `subsample`

Bagging/RF: sampling with replacement, size $= n$ (by default). Hence 0.632 / 0.368 I.4.

Boosting (subsample): sampling without replacement, a fraction 0.5-1.0. This is stochastic gradient boosting (Friedman).

Note [P]. The with/without-replacement distinction is a detail people mix up constantly. Bagging needs replacement to get different same-size samples; boosting just takes a fraction of the rows.

[P.8] Feature importance: the formulas and their biases

MDI (mean decrease in impurity), a.k.a. gain in XGBoost:

Imp (X_{j}) = \frac{1}{N} trees \sum t : node t splits on X_{j} \sum \frac{n _{t}}{n} Δ I (t)

Symbols. $Δ I (t)$ - the impurity decrease (or Gain) at node $t$ ; $n_{t} / n$ - the share of samples passing through the node (coverage weighting).

Importance types in XGBoost:

Type	Formula / meaning
`weight`	how many times the feature was used in splits
`gain`	average Gain over that feature's splits
`cover`	average hessian sum $H$ in nodes using the feature
`total_gain`, `total_cover`	totals rather than averages

Permutation importance:

Imp_{j} = err (data with X_{j} shuffled) - err (original data)

The biases (must know).

MDI is biased toward high-cardinality features: more candidate thresholds => more chances to stumble on a good split I.2.
Correlated features split the credit: the tree picks one, the other gets zero.
Permutation breaks under correlation: shuffling creates unrealistic samples outside the data's support.
An API trap: Booster.get_score() defaults to weight, while the sklearn wrapper's feature_importances_ in modern versions defaults to gain. One model, different rankings. Always set importance_type explicitly.

The main warning. Feature importance is not a causal effect. The model exploits any association, proxy, or confounder. It's the same trap as confounding in statistical inference, wearing an importance costume.

Where else it shows up. Direct parallel: a regression coefficient on observational data is an association, not an effect.

[P.9] Metrics: what measures what

ROC-AUC (the interpretation that matters):

AUC = P (s (x^{+}) > s (x^{-}))

That is the probability that a randomly chosen positive scores higher than a randomly chosen negative. Equivalent to the normalized Mann-Whitney $U$ statistic:

AUC = \frac{U}{n _{+} n _{-}}

Symbols. $s (\cdot)$ - the model's score; $n_{+}, n_{-}$ - the number of positives and negatives.

Consequence. AUC depends only on the ordering, not the score values. Therefore monotone calibration does not change AUC. An excellent interview fact.

Brier score (calibration + discrimination):

Brier = \frac{1}{n} i = 1 \sum n (p_{i} - y_{i})^{2}

Literally MSE on probabilities. A proper scoring rule.

Log-loss:

LogLoss = - \frac{1}{n} i \sum [y_{i} lo g p_{i} + (1 - y_{i}) lo g (1 - p_{i})]

Which when.

you need ranking => ROC-AUC; under heavy imbalance PR-AUC is more honest;
you need probabilities => log-loss, Brier, calibration curves.

Note [P]. Discrimination and calibration are different qualities. A model can rank perfectly (AUC = 0.95) and lie in its probabilities. If decisions are made at a monetary threshold - calibration is mandatory.

Calibration of tree models.

Random Forest - typically under-confident (averaging pulls probabilities toward the middle).
Boosting on log-loss - typically over-confident (pushes toward 0 and 1). Treatment: Platt scaling (a logistic layer on top of the score) or isotonic regression, on a separate set.

[P.10] `scale_pos_weight` and the base-rate shift

g_{i} \leftarrow ω g_{i}, h_{i} \leftarrow ω h_{i} for y_{i} = 1, ω \approx \frac{n _{-}}{n _{+}}

where $ω$ is scale_pos_weight.

What it does. Re-weights the positive class by multiplying its gradients and hessians.

What it breaks. Calibration. Exactly like undersampling: it shifts the base rate, so the predicted probabilities stop matching the real-world frequency.

How to choose.

you need ranking (top-N for manual review) => scale_pos_weight is fine, calibration irrelevant;
you need probabilities (threshold decisions in money) => better not to re-weight and tune the threshold instead; or re-weight and then calibrate.

Where else it shows up. The same story as "negative sampling breaks calibration": sampling/re-weighting moves the intercept (base rate), not the slopes. For inference (coefficients, odds ratios) it's nearly harmless; for prediction with probabilities it's critical.

[P.11] Histogram subtraction

hist (parent) = hist (left) + hist (right) ⟹ hist (big child) = hist (parent) - hist (small child)

The idea. Build the histogram only for the smaller child; get the bigger one by subtraction. Nearly a 2x saving.

Where else it shows up. A standard trick in every histogram-based booster (XGBoost hist, LightGBM).

Note [P]. A good "engineering" fact - shows you've read beyond the math.

[P.12] Weighted quantile sketch: why the quantiles are hessian-weighted

Define the rank function for feature $k$ :

r_{k} (z) = \frac{1}{\sum _{(x, h) \in D_{k}} h} (x, h) \in D_{k}, x < z \sum h

Candidate thresholds ${s_{k 1}, ..., s_{k l}}$ are chosen so that

∣ ∣ r_{k} (s_{k, j}) - r_{k} (s_{k, j + 1}) ∣ ∣ < ε ⟹ roughly 1/ ε candidates

Symbols. $D_{k} = {(x_{1 k}, h_{1}), ..., (x_{nk}, h_{n})}$ - pairs of (feature value, hessian); $ε$ - the precision (sketch_eps).

Why the weights are exactly $h_{i}$ . Because after Taylor the objective is equivalent to a weighted least squares with weights $h_{i}$ T.7. So a sample's importance for threshold selection is proportional to its hessian. The grid is built so that the total $h$ per bucket is equal, not the sample count.

Sanity check on MSE. $h_{i} = 1$ => weighted quantiles degenerate into ordinary ones. A special case, again.

Note [P]. "Why are the quantiles weighted?" is a marker question separating people who read the paper from people who read a blog post.

That was the practice tier. If one of these knobs just explained a bug you shipped last month - you know which reaction button to press ;)

New deep-dives (plus the provider discounts and free tiers I run into) land on my Telegram first: FitHappensML. Or catch me on X / Threads - ML/DS folks and app builders get a follow back.

Next up, part 3: the full derivations behind all of this, plus the cheat-sheet appendices - Part 3 - the Theory tier: full derivations and the cheat sheets

Every tree and boosting formula for ML interviews. Part 1: the core ten

Fit Happens ML — Thu, 09 Jul 2026 06:49:32 +0000

July 2026. Math doesn't rot, but library details do - XGBoost version notes below reflect 2.x-3.1.

The whole series:

Part 1 - the Interview tier: trees, forests, and the ten core formulas (you are here)
Part 2 - the Practice tier: the knobs you actually turn
Part 3 - the Theory tier: full derivations and the cheat sheets

I've been preparing for ML interviews, and the formulas for trees and boosting are scattered across ESL chapters, the XGBoost paper, documentation and a hundred blog posts, each with its own notation, each explaining the part the author happened to care about. At some point I gave up looking for one complete reference and made it myself.

While collecting I noticed my notes kept mixing three different kinds of "know", so I sorted everything into three tiers. For myself, originally, but the split turned out to be the most useful part:

[I] Interview - what I want to reproduce cold, at a whiteboard, mid-stress.
[P] Practice - the knobs I actually turn, and the places I've been burned.
[T] Theory - full derivations and cross-connections, for when a paper or an interviewer goes one level deeper.

Every formula follows the same template: symbols and dimensions => where it comes from => step by step => does anything get centered => where else this shows up => a note. The "where else" parts are, to me, the most valuable: the same three or four quantities keep reappearing across trees, logistic regression and statistics, and once you see that, the whole thing stops being a pile of facts.

The file came out so long that dev.to's editor waved a white flag, so it's a three-part series now. Cross-references like T.7 are section codes; I. sections live in this part, P. in part 2, T. in part 3, and every section carries its label in the heading, so Ctrl-F jumps you there. And it took a real amount of time to put together, so if you find it useful, leave a reaction: it genuinely raises the motivation to write more of these.

Section 0. Notation and conventions used throughout

0.1 Basic symbols and dimensions

Symbol	What it is	Dimension
$n$	number of samples	scalar
$d$	number of features	scalar
$K$	number of classes	scalar
$X$	design matrix	$n \times d$
$x_{i}$	features of sample $i$	column vector $d \times 1$
$y_{i}$	target of sample $i$	scalar (or $K$ -vector for multiclass)
$y^_{i}$ , $F (x_{i})$	model prediction (in margin space, before the link)	scalar
$p_{i}$	predicted probability (after the sigmoid)	scalar in $(0, 1)$
$g_{i}$	gradient of the loss w.r.t. the prediction	scalar; vector $g \in R^{n}$
$h_{i}$	hessian of the loss w.r.t. the prediction	scalar; vector $h \in R^{n}$
$T$	number of leaves in a tree	scalar
$w_{j}$	weight (prediction) of leaf $j$	scalar; vector $w \in R^{T}$
$q (\cdot)$	tree structure: sample => leaf index	$R^{d} \to {1.. T}$
$I_{j}$	set of samples in leaf $j$	subset of ${1.. n}$
$G_{j} = \sum_{i \in I_{j}} g_{i}$	sum of gradients in a leaf	scalar
$H_{j} = \sum_{i \in I_{j}} h_{i}$	sum of hessians in a leaf	scalar
$η$	learning rate / shrinkage	scalar
$λ, α, γ$	L2, L1, per-leaf penalty	scalars
$N$	number of trees in the ensemble	scalar

Do not confuse. $T$ is the number of leaves, not the depth and not the number of nodes. $H_{j}$ is the sum of hessians, not the number of samples - they coincide only for MSE, where $h_{i} = 1$ .

0.2 What gets centered and what doesn't - the once-and-for-all answer

A frequent source of confusion, because formulas from different models look alike.

Object	Centered?	Why
Features for a tree	No	A tree is invariant to any monotone transformation: the split "x < t" doesn't change under shift/scale. Centering, standardizing, logging - useless (numerical trivia aside)
Features for RF / GBM / XGBoost	No	Same reason. Hence "trees don't need feature scaling"
$G_{j}, H_{j}$ in XGBoost	No	These are sums of gradients and hessians, not moments of a distribution
$X^{⊤} W X$ (Fisher information in logistic regression)	No	This is the curvature of the log-likelihood, not a covariance of features; the intercept column centers implicitly
$S_{x y} = \sum (x_{i} - x ˉ) (y_{i} - y ˉ)$ (CUPED / OLS scatter)	Yes, mandatory	This is an actual covariance; without centering it stops being one

Mnemonic: centering belongs where you compute a covariance/variance of features. Where you compute loss curvature or sums of gradients, centering is beside the point.

One more difference from linear models: the design matrix for a tree needs no column of ones - a tree has no intercept; its role is played by $F_{0}$ (see T.8).

TIER 1 - [I] INTERVIEW

The ten I keep coming back to. Everything else here is built on top of them.

First, the thirty-second version of what a decision tree even is, so the formulas have something to sit on. A tree is a stack of if-else questions about features: is age < 34, is income < 52k. Each question splits the data in two, and the tree picks the question that makes the two halves as pure as possible, as close to single-class as it can manage. It grows greedily, one best question at a time, top to bottom. That's the entire machine. Everything below is just making "as pure as possible" precise, and then fixing the problems that greed creates.

The purity measures come first, and they have a surprisingly good pedigree: one is borrowed from a 1912 Italian statistician, the other from Claude Shannon.

[I.1] Impurity: Gini and entropy

Gini (t) = 1 - k = 1 \sum K p_{k 2}, Entropy (t) = - k = 1 \sum K p_{k} lo g p_{k}

The absolute classic: the first thing anyone learns about trees, and still the first thing interviewers reach for. These are the two standard answers to "how mixed is this node": the tree plugs one of them into information gain I.2 to score every split it ever makes.

Symbols. $t$ - a tree node; $p_{k}$ - the share of class- $k$ samples in node $t$ (scalar, $\sum_{k} p_{k} = 1$ ). Both quantities are scalars.

Where they come from.

Gini - the probability of mislabeling a sample if you assign it a class drawn at random from the node's own distribution: $\sum_{k} p_{k} (1 - p_{k}) = 1 - \sum_{k} p_{k 2}$ . And yes, it's named after that Gini (Corrado Gini of income-inequality fame), though it's not the inequality coefficient (that one is the Lorenz-curve construction). It's a categorical cousin from the same 1912 work, later reborn in ecology as the Gini-Simpson diversity index; both ask how likely two random draws are to disagree. For a 0/1 label the kinship becomes exact: the mean absolute difference between two random draws is $2 p (1 - p)$ , literally the binary impurity below.
Entropy - Shannon, 1948: the average number of bits (with $lo g_{2}$ ) needed to encode the class. So the other half of the tree runs on information theory. XGBoost and sklearn default to the natural log where one is needed.

Extremes. Both are zero in a pure node (single class). Maximum at the uniform distribution: Gini $= 1 - 1/ K$ , entropy $= lo g K$ .

Binary case (worth memorizing).

Gini = 2 p (1 - p), Entropy = - p lo g p - (1 - p) lo g (1 - p)

Centering. Not applicable - these are functions of class shares.

Where else it shows up. $p (1 - p)$ inside Gini is the Bernoulli variance - the same quantity as the hessian of the logistic loss I.10 and the weight in the Fisher information matrix $W$ of logistic regression. Not a coincidence: both measure "uncertainty" at a point.

Note [I] - why they're basically the same thing. Take entropy, $\sum_{k} p_{k} (- ln p_{k})$ , and replace the log with its first-order approximation $- ln p \approx 1 - p$ . You get $\sum_{k} p_{k} (1 - p_{k})$ , which is exactly Gini. Gini is entropy with the logarithm linearized: same shape (zero on pure nodes, maximal at uniform, both concave), just a hair less sharp near uniform distributions and cheaper to compute, because no log. Which is also the honest answer to the classic trap question "which one is better?": on real data the chosen splits almost never differ, and sklearn defaults to Gini for the boring reason: speed.

[I.2] Information Gain - the CART split criterion

IG = I (parent) - [\frac{n _{L}}{n} I (left) + \frac{n _{R}}{n} I (right)]

CART is Classification and Regression Trees - Breiman, Friedman, Olshen and Stone, 1984, and it arrived as a book, not a paper. It's one of the two family lines behind every modern tree: Quinlan's ID3/C4.5 line preferred entropy and multiway splits; the CART line preferred Gini and strictly binary splits. What's in sklearn and every boosting library today is essentially CART with forty years of engineering on top.

Symbols. $I (\cdot)$ - any impurity (Gini, entropy, MSE); $n_{L}, n_{R}$ - sample counts in the left and right child; $n = n_{L} + n_{R}$ . All scalars.

Where it comes from. "How much did heterogeneity drop after the split." The weights $n_{L} / n$ and $n_{R} / n$ exist because a child's impurity must be weighted by its size: a split that carves off a single sample into a pure leaf should not count as good.

Step by step (how CART picks a split).

For each feature $j = 1.. d$ , sort the values $x_{ij}$ .
For each candidate threshold $t$ (usually midpoints between adjacent unique values), partition into $L = {i : x_{ij} < t}$ and $R$ , compute IG.
Pick the pair $(j^{*}, t^{*})$ with maximal IG.
Recurse into the children.

Complexity. $O (n d)$ per node after sorting (or $O (n d lo g n)$ including the sort; XGBoost amortizes sorting via column blocks).

For regression. $I = MSE = \frac{1}{n _{t}} \sum_{i \in t} (y_{i} - y ˉ_{t})^{2}$ , and maximizing IG is equivalent to maximizing the reduction in the sum of squares.

Centering. Not needed: thresholds are compared within a feature, and monotone transformations don't change the ordering.

Where else it shows up. The XGBoost Gain formula I.8 is the exact same pattern - "left score + right score − parent score" - except the score is derived from the loss function instead of postulated.

Note [I]. The algorithm is greedy - locally best split, never revisited. Finding the globally optimal tree is NP-hard. Say this out loud at the interview. Second: the criterion is biased toward features with many unique values (more candidate thresholds => more chances to stumble on a good one). This is where the MDI bias in feature importance P.8 comes from.

Before the next formula - the one-paragraph version of Random Forest. A single deep tree is a great memorizer and a poor generalizer: nudge the training data and you get a completely different tree (that's high variance, I.9). Leo Breiman's fix was blunt: grow a lot of deep trees on bootstrap copies of the data (bagging, 1996), also force every split to choose from a random subset of features (random forests, 2001), and average the whole thing. He did this in his seventies, after a career as a probability theorist and years of freelance consulting. And "Random Forests" is, delightfully, a registered trademark he and Adele Cutler licensed out. The next formula is the entire mathematical justification for all that averaging.

[I.3] Variance of a mean of correlated variables - why RF decorrelates

Var (\frac{1}{N} i = 1 \sum N X_{i}) = ρ σ^{2} + \frac{1 - ρ}{N} σ^{2}

Symbols. $N$ - number of trees; $σ^{2}$ - variance of a single tree's prediction; $ρ$ - pairwise correlation of tree predictions. All scalars.

Where it comes from. Direct computation:

Var (\frac{1}{N} \sum X_{i}) = \frac{1}{N ^{2}} ⎣ ⎡ i \sum Var (X_{i}) + i \neq = j \sum Cov (X_{i}, X_{j}) ⎦ ⎤ = \frac{1}{N ^{2}} [N σ^{2} + N (N - 1) ρ σ^{2}]

= \frac{σ ^{2}}{N} + \frac{N - 1}{N} ρ σ^{2} = ρ σ^{2} + \frac{1 - ρ}{N} σ^{2}

How to read it (this is the point). Two terms:

$\frac{1 - ρ}{N} σ^{2}$ => goes to zero as trees are added. Averaging kills it.
$ρ σ^{2}$ => a floor that no amount of extra trees gets you below.

This is all of Random Forest. Since the floor is set by $ρ$ , you must reduce correlation between trees. Bootstrap gives part of the decorrelation; random feature selection at each split (max_features, mtry) gives most of it. If the data has one dominant feature, then without feature subsampling every tree picks it for the first split and the trees become nearly identical ( $ρ \to 1$ ) - and averaging buys almost nothing.

Centering. Not applicable.

Where else it shows up. The same formula is the foundation of any ensembling and of portfolio diversification in finance (risk of a portfolio of $N$ correlated assets). Literally one piece of math.

Note [I]. The single most important formula about random forests. If you can derive it and explain that max_features targets specifically the first term (the $ρ$ ), you're already above the median candidate who says "the forest adds randomness for diversity". It also answers why a forest doesn't overfit with more trees: growing $N$ only shrinks the second term; there's nothing for it to hurt.

[I.4] Bootstrap: the out-of-bag fraction

P (sample not drawn into the bootstrap) = (1 - \frac{1}{n})^{n} n \to \infty e^{- 1} \approx 0.368

Symbols. $n$ - training set size, which is also the bootstrap sample size (sampling with replacement, same size).

Where it comes from. The probability of not picking a given sample in one draw is $1 - 1/ n$ . There are exactly $n$ independent draws => $(1 - 1/ n)^{n}$ . The limit is the classic $e$ limit.

Consequences.

A bootstrap contains about 63.2 percent unique samples ( $1 - e^{- 1}$ ); the rest are duplicates.
About 36.8 percent of samples are out-of-bag (OOB) for a given tree.

How the ensemble's OOB estimate is computed (step by step).

For sample $i$ , find all trees whose bootstrap did not include it (about $0.368 N$ of them).
Predict $i$ by averaging/voting over those trees only.
Repeat for all $i$ ; compare with true labels => OOB error.

The key point. OOB is not an estimate for a single tree. Each sample is predicted by its own "honest sub-forest" that never saw it. That's why OOB ≈ cross-validation, for free.

OOB bias. You're evaluating a forest of roughly $0.368 N$ trees, not $N$ , so OOB is slightly pessimistic. With large $N$ (hundreds) the difference vanishes. With small $N$ (dozens) OOB is noisy and noticeably low - use regular CV there.

Where else it shows up. Bootstrap as an uncertainty tool (bootstrap CIs), and the ".632 rule" in bootstrap error estimates (the .632 and .632+ estimators).

Note [I]. OOB does not work for boosting - trees there are sequential and dependent; an "honest sub-ensemble that never saw the sample" doesn't exist. For boosting: a held-out set and early stopping. This makes an excellent bagging-vs-boosting comprehension check.

And now boosting: the other way to combine trees, and for tabular data the one that ate the world. Instead of averaging many independent deep trees, you build shallow ones sequentially, each correcting what the ensemble so far gets wrong. The XGBoost paper contains my favorite measure of how completely this took over: of the 29 winning Kaggle solutions published in 2015, 17 used XGBoost. A decade later, gradient-boosted trees still routinely beat deep learning on tabular benchmarks: an awkward fact for a deep-learning decade, and the reason this half of the file is the longer one.

[I.5] The additive model of boosting (forward stagewise)

y^_{i (t)} = y^_{i (t - 1)} + η f_{t} (x_{i}), y^_{i (0)} = F_{0}

The final model:

y^_{i} = F_{0} + η k = 1 \sum N f_{k} (x_{i})

Symbols. $f_{t}$ - the tree built at step $t$ (a map $R^{d} \to R$ ); $η$ - learning rate; $F_{0}$ - the constant intercept (see T.8).

The key property - forward stagewise. At step $t$ we freeze everything built so far and do not re-optimize earlier trees. This is what distinguishes boosting from a neural net, where all weights move together.

What a tree predicts. Not the target - a correction to the ensemble's current prediction. Everything follows from this: that a leaf's value is a learned parameter (not "the average of y"), and that it can therefore be regularized.

A tree as a map into leaf weights.

f_{t} (x) = w_{q (x)}, q : R^{d} \to {1.. T}, w \in R^{T}

where $q$ is the structure (which features, which thresholds) and $w$ is the vector of leaf weights.

Centering. Not applicable.

Where else it shows up. Forward stagewise additive modeling is the general frame: AdaBoost falls out of it under the exponential loss T.9, gradient boosting under an arbitrary differentiable one.

Note [I]. It follows that boosting cannot be parallelized across trees - they are sequential by construction. Parallelism lives inside the construction of a single tree (across features, across histogram bins). A frequent question.

[I.6] Pseudo-residuals: gradient boosting as descent in function space

r_{i t} = - [\frac{\partial L ( y _{i} , F ( x _{i} ) )}{\partial F ( x _{i} )}] F = F t - 1 = - g_{i}

Symbols. $L$ - the loss; $F_{t - 1}$ - the current ensemble; $r_{i t}$ - the pseudo-residual of sample $i$ at step $t$ . A vector $r \in R^{n}$ .

Where it comes from. Friedman's idea: we want to move in the direction that decreases the loss, but our parameters are not numbers - they're functions. Then the "steepest descent direction" in function space is the negative gradient of the loss with respect to the function's values at the training points. The tree is trained to approximate that negative gradient.

Sanity check on MSE. $L = \frac{1}{2} (y - F)^{2} \Rightarrow \frac{\partial L}{\partial F} = F - y \Rightarrow r_{i} = y_{i} - F_{t - 1} (x_{i})$ - ordinary residuals.

This is where the popular phrase "boosting fits the residuals" comes from. It is exact only for the squared loss; for every other loss, residuals are replaced by the negative gradient.

Sanity check on log-loss. $g_{i} = p_{i} - y_{i} \Rightarrow r_{i} = y_{i} - p_{i}$ - the gap between the label and the predicted probability.

Centering. Not applicable (gradients are already "deviations").

Where else it shows up. Direct analogy with ordinary gradient descent $θ \leftarrow θ - η \nabla_{θ} L$ ; here it's $F \leftarrow F + η \cdot (tree approximating - \nabla_{F} L)$ .

Note [I]. The sentence "gradient boosting is gradient descent in function space" is mandatory vocabulary. The next level of depth: XGBoost adds the second Taylor term and turns this into Newton's method T.2.

[I.7] XGBoost: the optimal leaf weight

w_{j *} = - \frac{G _{j}}{H _{j} + λ}

(out loud: "minus the sum of gradients over the leaf's samples, divided by the sum of their hessians plus lambda")

Symbols and dimensions. $G_{j} = \sum_{i \in I_{j}} g_{i}$ - scalar; $H_{j} = \sum_{i \in I_{j}} h_{i}$ - scalar; $λ$ - the L2 coefficient (reg_lambda, default 1); $w_{j *}$ - scalar, the prediction of leaf $j$ (in margin space!).

Where it comes from (short version; full derivation in **T.3).** Second-order Taylor => regroup the sum by leaves => inside a leaf you get a parabola $G_{j} w + \frac{1}{2} (H_{j} + λ) w^{2}$ => take the minimum the high-school way.

Three interpretations (be able to say all three).

Naive. Numerator - "how much error has accumulated in this leaf"; denominator - "how confidently we may correct it" (curvature) plus regularization.
Newtonian. A single sample's individual Newton step is $- g_{i} / h_{i}$ . Then

w_{j *} = \frac{\sum _{i \in I_{j}} h _{i} \cdot ( - \frac{g _{i}}{h _{i}} )}{\sum _{i \in I_{j}} h _{i} + λ}

That is a weighted average of individual Newton steps with weights $h_{i}$ , shrunk by $λ$ . The leaf trusts the samples with the most curvature.

Weighted least squares. See T.7.

Sanity check on MSE. $g_{i} = y^_{i} - y_{i}$ , $h_{i} = 1$ , $λ = 0$ :

w_{j *} = - \frac{\sum ( y ^ _{i} - y _{i} )}{∣ I _{j} ∣} = mean of the residuals in the leaf

The mean of residuals! So the intuition "a leaf's value is computed automatically" is a special case. It breaks the moment the loss isn't MSE or $λ > 0$ .

How the L2 works. $λ$ sits in the denominator => the weight is pulled toward zero. And since the denominator can't drop below $λ$ , the small leaves get strangled hardest (where $H_{j} ≪ λ$ ) - exactly where the overfitting risk is highest. Large leaves ( $H_{j} ≫ λ$ ) are barely touched.

Centering. No. $G, H$ are sums of derivatives, not moments.

Where else it shows up. Ridge regression: $β^= (X^{⊤} X + λ I)^{- 1} X^{⊤} y$ - the same "regularizer in the denominator" structure. In logistic-regression IRLS the update for $β$ contains $(X^{⊤} W X + λ I)^{- 1}$ .

Note [I]. Classic question: "how can L2 penalize a tree if a tree has no coefficients?" Answer: the tree's parameters are the leaf weights $w_{j}$ (learned constant corrections); ridge penalizes exactly those.

[I.8] XGBoost: the Gain formula (split value)

Gain = \frac{1}{2} [\frac{G _{L 2}}{H _{L} + λ} + \frac{G _{R 2}}{H _{R} + λ} - \frac{( G _{L} + G _{R} ) ^{2}}{H _{L} + H _{R} + λ}] - γ

(out loud: "one half of left-leaf score plus right-leaf score minus parent score, minus gamma")

Symbols. $G_{L}, H_{L}$ - sums of gradients and hessians in the left child; same for $R$ . Parent: $G = G_{L} + G_{R}$ , $H = H_{L} + H_{R}$ . $γ$ - gamma / min_split_loss.

Where it comes from. From the structure score T.5: the quality of a structure is $Obj^{*} = - \frac{1}{2} \sum_{j} \frac{G _{j 2}}{H _{j} + λ} + γ T$ . Gain is (quality before) minus (quality after), accounting for the fact that a split adds exactly one leaf ( $Δ T = + 1$ ), i.e. we pay $γ$ .

The three terms in the bracket. Left child's score + right child's score − parent's score. The same pattern as Information Gain I.2, but the score is derived from the loss, not postulated.

Split rule. Split if $Gain > 0$ . So $γ$ is a break-even threshold.

How $γ$ went from a global penalty to a local rule. In the objective the penalty is $γ T$ (global, per whole tree). Since every split adds $Δ T = + 1$ , the local price of any split is $γ \cdot 1 = γ$ . Hence the " $- γ$ " in the formula. This is a genuinely beautiful moment and it lands great at interviews.

How $λ$ influences split selection. A larger $λ$ suppresses the contribution of leaves with small $H$ => splits that produce small/uncertain leaves become less profitable => the tree grows more slowly. So L2 acts twice: it shrinks weights I.7 and it suppresses splits.

Sanity check on MSE. $h_{i} = 1$ , $λ = 0$ , $γ = 0$ : the bracket becomes $\frac{G _{L 2}}{n _{L}} + \frac{G _{R 2}}{n _{R}} - \frac{G ^{2}}{n}$ - exactly the reduction in the sum of squares, the classical CART criterion for regression. A special case, again.

Computational trick. To evaluate Gain at any threshold, you only need $G_{L}, H_{L}$ (the parent's $G, H$ are known). So walking left to right along the sorted feature and accumulating prefix sums covers all thresholds in a single pass.

With L1. Replace $G$ with $T_{α} (G)$ - soft-thresholding T.4.

Note [I]. The pair I.7+I.8 is the core of XGBoost. If you can derive both in two minutes, XGBoost questions are closed.

[I.9] The bias-variance decomposition

E [(y - f^(x))^{2}] = σ^{2} irreducible noise + (E [f^(x)] - f (x))^{2} Bias^{2} + Variance E [(f^(x) - E [f^(x)])^{2}]

Symbols. $f (x)$ - the true function; $f^(x)$ - the trained model (random through the randomness of the training set); $σ^{2} = Var (ε)$ - noise variance. Expectations are over training sets.

Where it comes from. Add and subtract $E [f^]$ inside the square, expand; the cross term vanishes.

How it maps onto trees.

Model	Bias	Variance	Treatment
A single deep tree	low	very high	ensembles
Bagging / Random Forest	low (unchanged)	reduced by averaging	more trees, lower $ρ$ I.3
Boosting	reduced sequentially	grows with the number of trees	early stopping, shrinkage, regularization

The key contrast (say it explicitly).

Bagging attacks variance => trees are deep (low bias), unpruned, averaged. Extra trees don't hurt.
Boosting attacks bias => trees are shallow (weak learners), added sequentially. Extra trees overfit.

Centering. Not applicable.

Where else it shows up. Regularization in any model is "add bias, remove variance". L1/L2 in linear models, $λ$ in XGBoost, dropout in neural nets.

Note [I]. Extra depth: in deep networks the classic U-shaped picture breaks (double descent) - fine to mention, carefully, without derailing the conversation.

[I.10] The sigmoid, log-loss, and g, h for the logistic loss

p = σ (z) = \frac{1}{1 + e ^{- z}}, σ^{'} (z) = σ (z) (1 - σ (z)) = p (1 - p)

L (y, z) = - [y lo g p + (1 - y) lo g (1 - p)]

g_{i} = \frac{\partial L}{\partial z _{i}} = p_{i} - y_{i}, h_{i} = \frac{\partial ^{2} L}{\partial z _{i 2}} = p_{i} (1 - p_{i})

Symbols. $z_{i}$ - the raw prediction (margin, log-odds); $p_{i} = σ (z_{i})$ ; $y_{i} \in {0, 1}$ .

Where it comes from (step by step).

$\frac{\partial L}{\partial p} = - \frac{y}{p} + \frac{1 - y}{1 - p} = \frac{p - y}{p ( 1 - p )}$
$\frac{\partial p}{\partial z} = p (1 - p)$
Chain rule: $\frac{\partial L}{\partial z} = \frac{p - y}{p ( 1 - p )} \cdot p (1 - p) = p - y$ . Everything cancels - this cancellation is exactly why the logit parametrization is so convenient.
$\frac{\partial ^{2} L}{\partial z ^{2}} = \frac{\partial ( p - y )}{\partial z} = p (1 - p)$ .

The meaning of $h_{i} = p (1 - p)$ . It's the Bernoulli variance - "how unsure the model is". Maximum $0.25$ at $p = 0.5$ (at the decision boundary), tending to zero as $p \to 0$ or $1$ . So samples near the boundary weigh more - they're the ones carrying information about where to move the boundary.

Technical nuance. As $p \to 0, 1$ the hessian $\to 0$ => division blowup in $- G / (H + λ)$ . XGBoost clips the hessian from below by a constant around $1 0^{- 16}$ ; additionally max_delta_step bounds $∣ w_{j} ∣$ .

Centering. Not applicable.

Where else it shows up - the most important connection in this file. The same quantity $p (1 - p)$ is:

the Bernoulli variance in statistics;
Gini impurity (binary case $= 2 p (1 - p)$ ) I.1;
the diagonal of the matrix $W$ in the Fisher information of logistic regression: $Cov (β^) = (X^{⊤} W X)^{- 1}$ , with $W = diag (p_{i} (1 - p_{i}))$ ; dimensions: $X$ is $n \times p$ , $W$ is $n \times n$ , the result is $p \times p$ ;
the hessian in XGBoost.

It is one and the same curvature of the logistic likelihood. See T.7 on Newton boosting ≈ IRLS.

Note [I]. Dropping this connection at an interview instantly shows that the models in your head are connected, not filed in separate boxes. One of the strongest cards you can play.

A plug, but an on-topic one, since half of Tier 1 just turned out to be statistics in disguise: I built a statistics exam app, Stat Exam Pro. The main focus is medical statistics, but if you're prepping for DS interviews it's worth a look anyway: the topics overlap more than you'd expect, and seeing the same ideas from another field's angle is weirdly refreshing. Everything is in test format with thoroughly worked answers. The section I'm proudest of is Paradoxes & Intuition Traps - start there ;)

Next up, part 2: the knobs you actually turn, and where they bite - Part 2 - the Practice tier: the knobs you actually turn

Anthropic's Free Credit Window Closes April 17. Don't Miss It.

Fit Happens ML — Sun, 05 Apr 2026 15:26:32 +0000

So here's the good news - Anthropic is once again showing its legendary generosity. Or more precisely, they're giving away free API credits.

If you're on the 5x Max plan, congratulations - you've been awarded $100 in credits you'll burn through in approximately one ambitious afternoon of agentic coding.

The claim window closes April 17, 2026. However, once redeemed, the credits remain active for 90 days - so there's no pressure to burn through them immediately.

Now here's the catch: this isn't an incentive to sign up. The eligibility cutoff was April 3rd - if you weren't already a paying subscriber by then, you're out of luck. Signing up now won't get you in.

And here's the part that worries me: companies don't give away free money without a reason. My bet? Rate limit cuts are coming. The credits are a cushion to soften the blow.

BTW you can clime your credits right on https://claude.ai/settings/usage

AI Wrote Code But You Don't Have npm? Here's What to Do

Fit Happens ML — Wed, 04 Mar 2026 08:55:49 +0000

A practical guide covering the complete installation process through to your first functional project

TL;DR: This guide lays out everything about Node.js and npm in one place. If you're using AI to write code (hello, vibe coders!), you might end up with a project you need to build but no npm installed. Here's what to install, what alternatives you have, what goes wrong and how to fix it + plus some senior dev tips.

IMPORTANT!
If you don't want to install anything and want to build everything in the cloud, you can skip the first part of the article until the Cloud Building heading.

Introduction

Node.js is basically JavaScript running outside the browser — on your computer, a server, whatever.

npm (Node Package Manager) comes with Node.js and lets you:

Grab thousands of libraries off the shelf
Keep track of versions
Run build scripts
Publish your own packages

Methods for Installing Node.js on Windows

The selection of an installation method should be based on what you are doing. For isolated or one-time usage scenarios, the official installer is OK. For ongoing development activities, NVM or Chocolatey are much better.

Installation via Official Installer

Step 1: Downloading

Go to https://nodejs.org/
Download the LTS version (Long Term Support) — it's more stable
Run the .msi installer
Agree to everything.
npm is usually added to PATH automatically
Then you need to verify:

node --version
# Should output: v20.x.x or v18.x.x

npm --version
# Should output: 10.x.x

Why It's Good

Easy peasy... Download LTS, click "Next" a bunch of times — boom, you've got Node.js. Easy if you don't want to mess with setup.

Why It Sucks

Updating means downloading a whole new installer
Can't run different versions side by side
Stuck with what you installed

FOR VIBE CODERS: If AI generated code for a newer version of Node.js and you have an old one — there will be errors.
P.S. NVM solves this problem. Check section Installation via NVM.

Installation via Chocolatey

What's Chocolatey?
Windows package manager. Like apt-get on Linux. Install once, then everything via commands.

What You Need First

Get Chocolatey (if missing):

Run PowerShell as admin:

# Check if choco is installed
choco --version

# If not, install:
Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((New-Object System.Net.WebClient).DownloadString('https://community.chocolatey.org/install.ps1'))

Install Node.js

# Install latest LTS
choco install nodejs-lts -y

# Specific version
choco install nodejs --version=20.11.0 -y

# Update
choco upgrade nodejs-lts

# Uninstall
choco uninstall nodejs

Why It's Good

Big upsides:

Easy updates: choco upgrade nodejs-lts
One command to uninstall
Adds to PATH automatically
Install any program this way

Installation via Winget

Windows 10 (1809+) and Windows 11 have a built-in package manager — Winget. No additional installation needed.

Commands

# Search available versions
winget search nodejs

# Install LTS version
winget install OpenJS.NodeJS.LTS

# Install latest (non-LTS)
winget install OpenJS.NodeJS

# Update
winget upgrade OpenJS.NodeJS.LTS

# Uninstall
winget uninstall OpenJS.NodeJS

Installation via NVM (Recommended for Developers)

NVM (Node Version Manager) is the standard in the Node.js development world. Allows you to have multiple versions of Node.js simultaneously and switch between them with a single command.

Especially useful when:

Old project requires Node.js 16, while new one requires Node.js 20
Testing compatibility across different versions
Gradually updating the version

Option 1: NVM for Windows

Download from: https://github.com/coreybutler/nvm-windows/releases
Install nvm-setup.exe
Open a new terminal

NVM Commands (for Windows)

IMPORTANT! NVM commands for Linux/Mac do not work on Windows. NVM for Windows has different syntax.

IMPORTANT! For the nvm use command to work, the terminal must be run as Administrator, or Developer Mode must be enabled in Windows (Settings -> Privacy & security -> For developers). Otherwise, Windows will block symlink creation and the Node.js version won't switch.

# Install specific version (MUST specify!)
nvm install 20
nvm install 18
nvm install 20.11.0

# Install latest available version (NOT lts!)
nvm install latest

# View installed versions
nvm list

# Use a specific version
nvm use 20

# Note: nvm alias default does NOT work on Windows!
# The version selected via nvm use becomes active automatically

# Uninstall a version
nvm uninstall 18

Option 2: Volta (Modern Alternative)

FOR VIBE CODERS: Volta is a relatively new tool. It automatically detects the required Node.js version for a project (reads from package.json). Very convenient if you often download other people's projects.

# Install Volta
winget install Volta.Volta

# Install Node.js
volta install node

# Install specific version
volta install node@20

# Pin version for project
volta pin node@20

IMPORTANT for Windows: For Volta to work correctly, you must enable Developer Mode in Windows settings. This allows the utility to create symlinks without requesting administrator rights.

P.S. An extra benefit is that Volta is one of the fastest Node.js version managers.

Check It Works

⚠️ Don't skip this! 10 seconds now saves hours of debugging.

PowerShell or CMD

# Check versions
node --version
npm --version

# Shorter way
node -v
npm -v

# Where npm lives
npm config get prefix

If It Doesn't Work

Most common issue: system can't find Node.js. Quick fix:

Restart terminal — PATH needs a new window
Check PATH manually:

System → Advanced → Environment Variables
Look for:
C:\Program Files\nodejs\
or
%APPDATA%\npm

Reinstall with "Add to PATH" checked

Basic npm Commands

npm is your main tool for Node.js. These are the everyday commands you'll use constantly.

Starting a Project

# Create new package.json (interactively)
npm init

# Create with default settings
npm init -y

# Create with all parameters specified
npm init --yes

Run npm init -y and you get a basic package.json. Think of it like "new document" in Word — your project's starting point.

Installing Dependencies

# Install all dependencies from package.json
npm install
# or shorthand
npm i

# Install specific package
npm install <package-name>
npm i <package-name>

# Install as development dependency (devDependencies)
npm install <package-name> --save-dev
npm i <package-name> -D

# Install globally (available everywhere in system)
npm install -g <package-name>

# Install specific version
npm install lodash@4.17.21
npm i express@4.18.2

IMPORTANT!
Global packages (-g) will bite you in big projects. They create system-specific dependencies. Always install locally in your project.

Managing Packages

# Update all packages
npm update

# Remove package
npm uninstall <package-name>
npm remove <package-name>

# Show globally installed packages
npm list -g --depth=0

# Show project's installed packages
npm list

Running Scripts

# Run script from package.json
npm run <script-name>

# Run standard scripts (start, test, build)
npm start
npm test
npm run build

Useful Flags

# Install without saving to package.json
npm install <package> --no-save

# Install only production dependencies
npm install --production

# Show outdated packages
npm outdated

# Security audit
npm audit
npm audit fix

Cache and Cleaning

# Clear npm cache
npm cache clean --force

# Delete node_modules and reinstall
# Safe deletion in Windows (bypasses path length limit):
npx rimraf node_modules
npm install

Building a Project

FOR VIBE CODERS: Typical scenario: AI wrote a software, there's a package.json, and you don't know what to do. Answer: npm install → npm run build.

Typical package.json Structure

{
  "name": "my-project",
  "version": "1.0.0",
  "main": "dist/index.js",
  "scripts": {
    "build": "tsc",
    "dev": "tsc -w",
    "start": "node dist/index.js",
    "test": "jest",
    "lint": "eslint ."
  },
  "dependencies": {
    "express": "^4.18.0"
  },
  "devDependencies": {
    "@types/node": "^20.0.0",
    "typescript": "^5.3.0"
  }
}

Typical Build Scripts

# Install dependencies
npm install

# Build project
npm run build

# Development mode (watch - automatic rebuild)
npm run dev

# Run in development mode with nodemon
npm run dev:watch

Build Result

After npm run build:

TypeScript compiles to JavaScript
Files appear in the dist/ folder
Ready-to-use code can be deployed

P.S. dist/ is the standard name for the compiled code folder (from "distribution"). It's usually what goes to production.

Alternative Package Managers

npm is not the only option. Yarn and pnpm have their own advantages.

Yarn

# Enable corepack (built into Node.js)
corepack enable
# Activate stable yarn version
corepack prepare yarn@stable --activate

# Initialize project
yarn init

# Install dependencies
yarn
yarn add <package>
yarn add <package> -D

# Build
yarn build

Yarn advantages:

Faster than npm (caches packages)
Prettier console output
Built-in yarn.lock (equivalent to package-lock.json)

In my opinion, there's no major difference between npm and Yarn, you can choose what you like better.

pnpm

# Enable corepack and activate pnpm
corepack enable
corepack prepare pnpm@latest --activate

# Use
pnpm install
pnpm add <package>
pnpm run build

pnpm advantages:

Saves disk space (symlinks)
Faster than npm and yarn
Dependency isolation (better for monorepos)

IMPORTANT!
pnpm uses content-addressable storage. One physical file = many symbolic links. This really saves space — sometimes 3-4x.

Disadvantages: Not all projects work correctly with pnpm due to different node_modules structure.

Building in the Cloud

Need to build but Node.js isn't on the machine? Using someone else's computer? Cloud is the easy answer.

Option 1: GitHub Codespaces

Open repo on GitHub
Click "Code" → "Codespaces"
Spin up a new codespace
In the terminal:

npm install
npm run build

Plus: Free for personal use, nothing to install.

Option 2: StackBlitz

Head to https://stackblitz.com/
Create new project
Upload files (drag & drop)
Terminal:

npm install
npm run build

Option 3: Gitpod

Go to https://gitpod.io/
Paste your repo URL
Dev environment opens automatically

GitHub Actions for Automated Builds

IMO the simplest (and correct) way is CI/CD automation. You push code, GitHub itself builds the project and checks that nothing broke.

Why?

Build runs automatically on push
Can build under different Node.js versions
Artifacts are saved in Release
Automated tests

Workflow Example

Create file .github/workflows/build.yml:

name: Build

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  build:
    runs-on: ubuntu-latest

    strategy:
      matrix:
        node-version: [18.x, 20.x]

    steps:
    - uses: actions/checkout@v4

    - name: Use Node.js ${{ matrix.node-version }}
      uses: actions/setup-node@v4
      with:
        node-version: ${{ matrix.node-version }}
        cache: 'npm'

    - name: Install dependencies
      run: npm ci

    - name: Build
      run: npm run build

    - name: Upload artifacts
      uses: actions/upload-artifact@v4
      with:
        name: dist-${{ matrix.node-version }}
        path: dist/

P.S. Today, GitHub Actions is one of the most popular CI/CD tools. According to surveys, more than 70% of developers use it for build and test automation.

More Complex Example with Release

name: Release

on:
  push:
    tags:
      - 'v*'

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v4

    - name: Setup Node.js
      uses: actions/setup-node@v4
      with:
        node-version: '20'
        cache: 'npm'

    - name: Install and Build
      run: |
        npm ci
        npm run build

    - name: Create Release
      uses: softprops/action-gh-release@v1
      with:
        files: dist/**
      env:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

How to Run Build Locally (act)

SENIOR GOTCHA!
act requires Docker Desktop to be installed! Without Docker, the act command will just fail with an error. On Windows, Docker setup is a separate task, be prepared for this.

# Install act
choco install act -y

# Run workflow locally
act

You could say that act is "GitHub Actions locally." Run it — and you have your own personal CI/CD on your computer with blackjack and... oops.

Troubleshooting

90% of Node.js stuff fixes by restarting terminal, clearing cache, or switching versions. Here's the rest.

npm install slow

Most common issue, especially on Windows.

Fixes:

# Use npm ci instead (faster, uses lockfile)
npm ci

# Clear cache
npm cache clean --force

# Switch to faster manager
npm install -g pnpm
pnpm install

Out of memory

# Bump memory (PowerShell)
$env:NODE_OPTIONS = "--max_old_space_size=4096"
npm run build

Permission denied

Windows + perms = pain. Run terminal as admin.

Or move npm folder:

# Global packages in user dir (PowerShell)
mkdir "$env:USERPROFILE\npm-global"
npm config set prefix "$env:USERPROFILE\npm-global"
# Add to PATH

That's it! Covered the main stuff.

Thanks for reading! Subscribe, like, feedback = more good content. Thanks!

My 2026 Agent Workflow: How I Stopped Burning $200/Month on Claude Code, Antigravity & Codex

Fit Happens ML — Mon, 09 Feb 2026 06:44:54 +0000

I've watched devs in 2026 drop $150–200 a month on Claude Code or Antigravity, type "build me an app," and end up with 800 lines of hallucinated garbage plus a shiny "context limit reached" error. I did the same - burned through tokens like they were free beer at a conference happy hour, got nothing usable, and felt like an idiot.

Of course the hype is real: Claude Code (now the most popular coding agent, writing ~90% of Anthropic's own code with Opus 4.5), Antigravity (Google's agent-first IDE that plans autonomously across editor/terminal/browser), Codex (OpenAI's lightweight CLI agent that drafts PRs and runs tests). But "vibe coding" - casually chatting with the agent - fails hard because you're basically asking a very expensive intern to write production code without a spec.

The fix is discipline: a controlled loop instead of chaos. This pipeline is inspired by GitHub's Spec Kit but with more control - multi-model checks, MCP for fresh docs, rules to avoid repetition. Follow these 6 steps, and agents accelerate 3–5x instead of torching your budget.

Step 1: Spec First - Write spec.md Before Any Code

Don't feed the agent any code requests until you have a solid spec.md. Describe the goal, inputs/outputs, constraints, edge cases, and minimum tests. Use your favorite LLM to ask clarifying questions until everything's nailed down. Save it in the repo as spec.md.

Trust me, it'll save your ass - edge cases kill agents otherwise. Without them, Claude Code might choke on CSV quoting hell, Antigravity plans only happy paths, Codex hallucinates invalid inputs.

Example (Word Frequency CLI Analyzer):

# spec.md - Word Frequency CLI Analyzer (MVP)

## Goal
CLI tool that reads a text file (any encoding) and outputs:
- Total word count
- Number of unique words
- Top N most frequent words (default N=10)
- Optional: save results to JSON

Useful for quick text analysis (logs, notes, scraped data, chat exports).

## Inputs/Outputs
- Input: path to text file (required arg)
- Flags/Options:
  - --top N (int, default 10): show top N words
  - --json (flag): output to results.json instead of stdout
  - --min-length M (int, default 3): ignore words shorter than M chars
- Output:
  - stdout: human-readable summary + top list
  - or JSON file if --json

## Constraints
- Python 3.11+
- No external libraries (only stdlib: argparse, collections, json, pathlib, re)
- Handle files up to 100MB (stream reading, no full load into memory)
- Support UTF-8 + common encodings (utf-8, latin-1, cp1252 fallback)
- Case-insensitive counting (normalize to lower)

## Edge Cases
- Empty file → 0 words, empty top list
- File with only punctuation/spaces → 0 words
- Very large file (>50MB) → must stream, no OOM
- Mixed encodings → try utf-8 first, fallback to latin-1 or ignore errors
- Words with hyphens/apostrophes (e.g. "don't", "well-known") → count as one word
- Non-English text (accents, Cyrillic, emojis) → count words properly (split on whitespace + punctuation)
- Invalid path → clear error message + exit 1
- No args → show usage/help

## Tests (minimum set)
- Happy path: small English text file → correct counts & top words
- Empty file → "0 words found"
- File with only punctuation → "0 words found"
- Large-ish file (simulate 10MB lorem ipsum) → no crash, correct stats
- Non-UTF8 file (latin-1) → reads without crash
- JSON output → valid JSON with structure { "total_words": int, "unique_words": int, "top_words": list of [word, count] }

## Non-Goals (out of scope for MVP)
- GUI
- Real-time streaming from stdin
- Stemming/lemmatization
- Stop words filtering
- Multi-file support

Writing the spec sometimes feels like it takes as long as coding the thing yourself. True sometimes. But this is the investment: the skill of forcing AI to write good code is what pays in 2026. Writing everything by hand? That's dinosaur territory now.

Step 2: Plan & Tickets - Break It Down

Feed spec.md to a strong thinking model. Ask it to create a plan: 5–15 small stages, each deliverable in 1–3 senior dev days, each bringing real value (MVP-0 => killer feature => polish).

Then turn the plan into tiny tickets - one focused change per ticket.

Why it works: Big requests overload context and cause drift. Tiny ones keep models sharp and let you catch errors early. Update the plan every 3 stages when reality bites (Pro Tip: use Obsidian for epics and iterative refactoring).

Step 3: Rules & Context - Set Boundaries

Create rules.md (CLAUDE.md, GEMINI.md etc.) with lint rules, forbidden patterns, test commands, style guides. Agents reference it automatically - saves tokens and sanity (no more repeating "use black" 50 times).

Give context: which files to change, repo structure (connect GitHub in Codex/Claude Code), versions/arch bans, known pitfalls, team conventions.

Bonus: Use MCP (Model Context Protocol) - the 2026 standard (thousands of servers now) - to connect fresh library docs. Agents pull actual APIs instead of hallucinating outdated ones. Huge win.

Step 4: Agent Activation - Let Them Loose (Only Now)

Only now connect the repo and activate agents

Step 5: Tests & Multi-Model Check - Every Single Step

For each ticket: define tests in spec/plan, make the agent run them, fix failures.

Don't trust one agent: one codes, another reviews/refactors (Claude for code, Codex for PR suggestions, Antigravity for verification). Cross-check catches blind spots.

Treat agents like interns: clear plan + reviews = better output. Bonus: You'll level up your PM skills massively.

Step 6: Commit & Iterate - Save Points & Refactor

Commit after every successful ticket/test pass. Use branches for experiments - Git is your undo when an agent make sh * t.

Pro Tip: Refactor every 3–4 stages: ask an agent "evaluate current solution for architecture/best practices" while context is fresh.

Conclusion

In 2026, vibe coding isn’t about how fast you can type code anymore.

It’s about how well you can make AI write the code for you.

You’re no longer the senior dev hammering away at the keyboard for eight hours straight. You’re now part product manager, part architect, part tech lead - except your “team” costs $150–200 a month, never takes vacation, but will happily hallucinate garbage if you don’t give them crystal-clear tasks.

Quick Cheatsheet / Memo Spec => plan => rules & context => tiny tickets => agent execution => tests + multi-model cross-check => commit => refactor every few stages.

Thanks for reading!
More no-BS takes on AI agents, tools, and surviving the token apocalypse in my Telegram channel and on X (Twitter).

2x Limits Right Now - Here’s How Much You Can Actually Code Before They Cut It (Codex Promo)

Fit Happens ML — Tue, 03 Feb 2026 11:09:05 +0000

If you blinked yesterday (Feb 2, 2026), you missed it: OpenAI quietly launched a dedicated Codex app for macOS - their shiny new “command center” for AI coding agents. And to make sure everyone notices, they threw in a promo that feels like a holiday gift you didn’t ask for but kinda needed:

Free & Go users get temporary access to Codex (trial mode, limits unknown but “included”).
Paid plans (Plus, Pro, Business, Enterprise, Edu) get 2× rate limits for a “limited time” - no end date announced, so it could vanish tomorrow.

Check your dashboard or the Codex pricing page - if you see doubled numbers or a banner, congrats, you’re in the window.

What Actually Changed (Quick Numbers, No BS)

Limits are counted in messages/tasks over sliding 5-hour windows (local + cloud share the same pool - mix them and it burns faster). Plus code reviews per week.

Rough current promo numbers (from developers.openai.com table + recent announcements):

Plus ($20/mo)

Local messages: ~90–450 / 5h (was 45–225)
Cloud tasks: ~20–120 / 5h (was 10–60)
Code reviews: ~20–50 / week (was 10–25)

Pro ($200/mo)

Local: ~600–3000 / 5h (was 300–1500)
Cloud: ~100–800 / 5h (was 50–400)
Reviews: ~200–500 / week (was 100–250)

Business usually mirrors Plus, Enterprise scales with credits (no hard caps if flexible pricing).

Pro tip: Switch to GPT-5.1-Codex-Mini for local tasks - gets you up to ~4× more messages in the same window (Plus → ~360–1800 local, Pro → ~2400–12000). Cheap way to stretch the promo.

If you hit the wall anyway - buy credits (local ~5 credits/msg, cloud/review ~25). No upgrade needed.

How to Start

1) Open ChatGPT => hit “Try in your IDE” or go to openai.com/codex

2) Choose VS Code (or wait for native app install)

3) Install the “Codex - OpenAI’s coding agent” extension (openai.chatgpt)

4) Sign in with your ChatGPT account

5) Open your repo - Codex sees files, context, can diff, run tests, delegate to cloud

Sandbox mode keeps it safe (limits folder writes/network). If you’re paranoid - start with small local tasks.

P.S. Why This Launch Feels Like They Read Your Mind

Peak OpenAI move: new powerful app + extra allowance right when usage is exploding (they say Codex doubled since GPT-5.2-Codex in December, >1M devs touched it last month).

It’s like they’re saying: “We know you’ll delegate half your workload to agents this week - here’s double the rope.”

By the way - hurry “limited time” usually means “until we see usage spike and panic about costs.” So, grab the extra limits while they’re hot. May your agents ship code, not excuses. 🚀

Thanks for reading.

I’m building open-source stuff and writing about ML/AI in my Telegram channel: https://t.me/FitHappensML and on X/Twitter: https://x.com/FitHappensML

Building a Custom FB2 Converter for LLMs

Fit Happens ML — Thu, 29 Jan 2026 11:39:46 +0000

TL;DR:
I "vibecoded" a custom Python tool to convert legacy .fb2 e-books into structured Markdown (perfect for LLMs/RAG/Obsidian) and Plain Text. No ads, no bloat, just 150 lines of code.

The App: Live Streamlit Demo
The Code:
FitHappensML / fb2-to-md-converter

A lightweight CLI and Web tool to convert FB2 books to Markdown or Plain Text. Includes a built-in Streamlit reader and smart formatting support.
FB2 to TXT/MD Converter & Reader

🚀 Try the Live Demo: fb2-to-md-converter.streamlit.app

✍️ Read the Story behind the code: Vibecoding Your Way Out of Format Hell (Medium)

This project is a Python-based utility for converting .fb2 (FictionBook) files into .txt (plain text) or .md (Markdown). The tool offers two interfaces: a user-friendly web UI built with Streamlit for reading and converting, and a command-line interface (CLI) for fast processing and automation.

✨ Key Features
- Dual Interfaces
  
  🎨 Web UI (Streamlit): Upload your files, read books directly in the browser, and download the result in your desired format.
  
  ⚙️ Command-Line Interface (CLI): Quickly convert files from your terminal, perfect for scripting and batch processing.
- Smart Formatting: An option to convert FB2 tags (like subtitles and emphasis) into corresponding Markdown syntax.
- Dual Export Formats: Save your books as clean .txt or as formatted .md files.
- Built-in Reader…
View on GitHub

Hello, fellow builders!

If you're anything like me, you probably have a digital hoard of books. In my case, it's a massive collection of .fb2 (FictionBook) files. Solid format, xml-based, widely supported... until you need to feed it into a Large Language Model (LLM).

Here's the problem: LLMs eat text, not XML tags.

I needed to convert my entire library into clean, structured Markdown. I needed headers to actually be headers (### Chapter 1), and emphasis to be italics (*wow*), so the model understands the semantic structure of the narrative.

I looked at existing tools.

The Desktop Apps: Bloated, require installation, often Windows-only 90s relics.
The Online Converters: "Upload clean_code.fb2... waiting... Download your file after watching this 30s ad". No thanks.
The Scripts: Most just strip all tags blindly, turning a beautiful dialogue into a wall of text.

If you are a developer in 2026, you don't hunt for software. You vibecode it.

It is faster to tailor a bespoke suit of a script than to shop for ill-fitting off-the-rack solutions. Plus, when you build it, you own the pipeline.

So, I built my own FB2 to Markdown converter. It has a CLI for batch processing and a Streamlit UI because sometimes I just want to read a chapter in the browser.

Here is how I did it, and how you can do it too.

The Strategy: FB2 is just XML

Don't overcomplicate it. An FB2 file is just an XML file with a specific schema. We don't need a heavy e-book library; we need BeautifulSoup.

Here is the core logic. We parse the XML, look for specific tags (<subtitle>, <emphasis>), and map them to Markdown.

The Core Converter

I created a converter.py. The trick is handling nested tags. A paragraph <p> might contain <emphasis> inside it.

from bs4 import BeautifulSoup
from bs4.element import Tag

def _get_formatted_text(tag: Tag) -> str:
    """Recursively process tags to keep italics and bolding."""
    parts = []
    for item in tag.children:
        if isinstance(item, Tag):
            if item.name == 'emphasis':
                parts.append(f'*{item.get_text(strip=True)}*')
            else:
                parts.append(_get_formatted_text(item))
        else:
            parts.append(str(item))
    return "".join(parts)

This recursive function is the secret sauce. Instead of text_content() which flattens everything, this preserves the vibe of the text.

Then, the main loop allows us to choose between "Raw Text" and "Smart Formatting":

def convert_fb2_to_txt(fb2_content: str, smart_formatting: bool = False) -> str:
    soup = BeautifulSoup(fb2_content, 'lxml-xml')
    text_parts = []

    # Extract Metadata (Title, Author)
    description = soup.find('description')
    # ... extraction logic ...

    # Extract Body
    body = soup.find('body')
    for element in body.find_all(['p', 'subtitle', 'empty-line']):
        if element.name == 'p':
            if smart_formatting:
                 text_parts.append(_get_formatted_text(element).strip() + '\n\n')
            else:
                 text_parts.append(element.get_text(strip=True) + '\n\n')
        elif element.name == 'subtitle':
            # Boom: Semantic headers for the LLM
            text_parts.append(f"### {element.get_text(strip=True)}\n\n")

    return "".join(text_parts)

The Interface: Streamlit for Instant Gratification

I love CLI, but sometimes I want to drag-and-drop. Streamlit is perfect for this. It takes 5 minutes to build a UI that looks decent.

In app.py, I process the file and offer two flavors of download:

import streamlit as st
from converter import convert_fb2_to_txt

st.title("📖 FB2 Reader & Converter")

uploaded_file = st.file_uploader("Upload .fb2", type=['fb2'])

if uploaded_file:
    # Read the bytes
    content = uploaded_file.getvalue().decode('utf-8')

    # Dual Conversion
    plain_text = convert_fb2_to_txt(content, smart_formatting=False)
    markdown_text = convert_fb2_to_txt(content, smart_formatting=True)

    # The Reader View
    st.markdown(markdown_text)

    # Sidebar Downloads
    st.sidebar.download_button("Download .md", markdown_text, file_name="book.md")
    st.sidebar.download_button("Download .txt", plain_text, file_name="book.txt")

This gives me immediate visual verification. I can see if the <subtitle> tags are actually rendering as headers before I commit to converting my whole library.

The CLI: For the Serious Batching

Finally, cli.py. Because I'm not going to drag-and-drop 500 books.

import argparse
from converter import convert_fb2_to_txt

# ... setup format args ...

if args.format == 'md':
    result = convert_fb2_to_txt(content, smart_formatting=True)
else:
    result = convert_fb2_to_txt(content, smart_formatting=False)

Now I can just run:
python cli.py "War_and_Peace.fb2" -f md
And get a perfect Markdown file ready for RAG (Retrieval-Augmented Generation) or fine-tuning.

Why "Vibecode" it?

Could I have found a tool to do this? Probably.
Would it handle the specifics of <subtitle> tags nested in <section> blocks exactly how I wanted? No.

By spending an hour writing this, I now have a tool that is:

Fast: No uploads, no ads.
Private: My books stay on my machine.
Correct: The output is formatted exactly for my LLM's consumption.

In the era of AI, the ability to quickly whip up data transformation scripts is a superpower. Don't be afraid to reinvent the wheel if the tire on the existing wheel is flat.

Happy coding!

DEV Community: Fit Happens ML

Every tree and boosting formula for ML interviews. Part 3: theory

TIER 3 - [T] THEORY

[T.1] The regularized XGBoost objective

[T.2] The second-order Taylor expansion

[T.3] Regrouping by leaves and deriving the optimal leaf weight

[T.4] Soft-thresholding: how the L1 case is solved

[T.5] The structure score

[T.6] Deriving Gain from the structure score

[T.7] Equivalence to weighted least squares: Newton boosting ≈ IRLS

[T.8] Initializing F0: what the "zeroth tree" predicts

[T.9] AdaBoost: the exponential loss and the logit connection

[T.10] Multiclass: softmax, gradients, the diagonal hessian

[T.11] SHAP: Shapley values

[T.12] Functional gradient descent - the general frame

Appendix A. The cross-connections - "where else does this show up"

Appendix B. What gets centered - the quick card

Appendix C. The dimension checklist (self-test)

Appendix D. The 60-second checklist - what to say

Every tree and boosting formula for ML interviews. Part 2: practice

TIER 2 - [P] PRACTICE

[P.1] The leaf value: CART, GBM (line search), XGBoost

[P.2] Learning rate (shrinkage)

[P.3] min_child_weight is NOT a sample count

[P.4] max_delta_step

[P.5] Cost-complexity pruning and the number of leaves

[P.6] max_features / mtry - the heuristics

[P.7] Bootstrap size and subsample

[P.8] Feature importance: the formulas and their biases

[P.9] Metrics: what measures what

[P.10] scale_pos_weight and the base-rate shift

[P.11] Histogram subtraction

[P.12] Weighted quantile sketch: why the quantiles are hessian-weighted

Every tree and boosting formula for ML interviews. Part 1: the core ten

Section 0. Notation and conventions used throughout

0.1 Basic symbols and dimensions

0.2 What gets centered and what doesn't - the once-and-for-all answer

TIER 1 - [I] INTERVIEW

[I.1] Impurity: Gini and entropy

[I.2] Information Gain - the CART split criterion

[I.3] Variance of a mean of correlated variables - why RF decorrelates

[I.4] Bootstrap: the out-of-bag fraction

[I.5] The additive model of boosting (forward stagewise)

[I.6] Pseudo-residuals: gradient boosting as descent in function space

[I.7] XGBoost: the optimal leaf weight

[I.8] XGBoost: the Gain formula (split value)

[I.9] The bias-variance decomposition

[I.10] The sigmoid, log-loss, and g, h for the logistic loss

Anthropic's Free Credit Window Closes April 17. Don't Miss It.

AI Wrote Code But You Don't Have npm? Here's What to Do

Introduction

Methods for Installing Node.js on Windows

Installation via Official Installer

Step 1: Downloading

Why It's Good

Why It Sucks

Installation via Chocolatey

What You Need First

Install Node.js

Why It's Good

Installation via Winget

Commands

Installation via NVM (Recommended for Developers)

Option 1: NVM for Windows

NVM Commands (for Windows)

Option 2: Volta (Modern Alternative)

Check It Works

PowerShell or CMD

If It Doesn't Work

Basic npm Commands

Starting a Project

Installing Dependencies

Managing Packages

Running Scripts

Useful Flags

Cache and Cleaning

Building a Project

Typical package.json Structure

Typical Build Scripts

Build Result

[P.3] `min_child_weight` is NOT a sample count

[P.4] `max_delta_step`

[P.6] `max_features` / `mtry` - the heuristics

[P.7] Bootstrap size and `subsample`

[P.10] `scale_pos_weight` and the base-rate shift