Every tree and boosting formula for ML interviews. Part 2: practice

July 2026. Math doesn't rot, but library details do - XGBoost version notes reflect 2.x-3.1.

The whole series:

• Part 1 - the Interview tier: trees, forests, and the ten core formulas
• Part 2 - the Practice tier: the knobs you actually turn (you are here)
• Part 3 - the Theory tier: full derivations and the cheat sheets

This is the practice tier of the reference: the parameters you actually turn when training tree ensembles, and the places where the intuitive reading of a knob is simply wrong. Notation and the ten core formulas live in part 1; cross-references like I.7 point there, T.7 points to part 3.

---

TIER 2 - [P] PRACTICE

What you actually turn, and where people actually get burned.

---

[P.1] The leaf value: CART, GBM (line search), XGBoost

Three different answers to "what's inside a leaf".

CART (a plain tree). Computed "automatically":

$$w_j = \bar y_{I_j} \ \text{(regression)}, \qquad w_j = \arg\max_k p_k \ \text{(classification)}$$

GBM (Friedman) - line search. The tree is grown by the ordinary CART criterion on pseudo-residuals, and then the leaf values are re-fitted by one-dimensional optimization inside each leaf:

$$\gamma_{jt} = \arg\min_{\gamma} \sum_{i \in I_j} L\big(y_i, F_{t-1}(x_i) + \gamma\big)$$

XGBoost - a closed formula. $w_j^\ast = -G_j/(H_j+\lambda)$ , see I.7. This is the analytic solution of the same problem after the quadratic approximation.

Note [P]. Three stages of evolution: "the mean" => "one-dimensional search" => "a closed formula from Taylor". A good answer to "how does XGBoost differ from GBM".

Trap. For reg:absoluteerror (MAE) and quantile regression the hessian is constant, the Newton step degenerates => XGBoost uses adaptive trees: it builds the structure, then recomputes the leaf values as the median (or the quantile). In other words, it falls back to line-search logic.

---

[P.2] Learning rate (shrinkage)

$$\hat y^{(t)} = \hat y^{(t-1)} + \eta f_t(x), \qquad \eta \in (0, 1]$$

Role. Shrinking each tree's contribution leaves "room" for the following ones. A form of regularization.

The trade-off. $\eta$ and the number of trees $N$ trade against each other: roughly $\eta \cdot N \approx \text{const}$ for comparable fit. Smaller $\eta$ + larger $N$ => better generalization, longer training.

Practice. $\eta \in [0.01, 0.1]$ ; the number of rounds is decided by early stopping, not by hand.

Note [P]. The standard recipe: fix $\eta = 0.05$ , tune everything else, then at the end cut $\eta$ by 2-4x and raise the round cap proportionally.

---

[P.3] min_child_weight is NOT a sample count

$$\text{a split is rejected if } H_{\text{child}} = \sum_{i \in I_{\text{child}}} h_i < \text{min child weight}$$

The $g$ / $h$ table by objective (memorize):

Objective	$g_i$	$h_i$	What min_child_weight means
reg:squarederror, $L=\frac12(y-\hat y)^2$	$\hat y_i - y_i$	$1$	number of samples in the leaf
binary:logistic	$p_i - y_i$	$p_i(1-p_i)$	total uncertainty
count:poisson (log link)	$e^{\hat y_i} - y_i$	$e^{\hat y_i}$	sum of expected intensities
reg:absoluteerror	$\text{sign}(\hat y_i - y_i)$	$1$	number of samples

The intuition-breaking consequence. In binary classification, a leaf of ten confident samples ( $p\approx 0$ or $1$ , $h\approx 0$ ) has a tiny $H$ and will be rejected. A leaf of three borderline samples ( $p\approx0.5$ , $h=0.25$ ) gives $H = 0.75$ and may pass. The parameter doesn't say "few samples" - it says "not enough reliable information".

Scaling. $H$ grows with data size => the default of 1 restricts almost nothing at a million rows. Tune it with $n$ in mind.

Note [P]. A top-3 XGBoost trap question. Plenty of people confidently answer "minimum number of samples in a leaf" - true only for MSE.

---

[P.4] max_delta_step

$$|w_j| \le \text{max delta step}$$

Why. Under extreme imbalance in logistic tasks $h_i = p(1-p) \to 0$ , the denominator $H+\lambda$ is tiny => the leaf weight blows up => unstable updates. A hard cap (typically 1-10) makes the step conservative.

Note [P]. The standard recommendation for heavily imbalanced binary classification, together with scale_pos_weight P.10. Default 0 = no cap.

---

[P.5] Cost-complexity pruning and the number of leaves

Classic CART (post-pruning):

$$R_\alpha(T) = R(T) + \alpha|\tilde T|$$

Symbols. $R(T)$ - the tree's training error; $|\tilde T|$ - the number of leaves; $\alpha \ge 0$ - the price of one leaf. As $\alpha$ grows, the optimal subtree shrinks (a nested sequence of subtrees exists).

XGBoost - the same idea, but inside the objective:

$$\Omega(f) = \gamma T + \frac12\lambda\sum_{j=1}^{T} w_j^2 + \alpha_{L1}\sum_{j=1}^T |w_j|$$

Here $\gamma$ plays the role of CART's $\alpha$ .

How $T$ relates to splits. A binary split turns one leaf into two: $\Delta T = +1$ . Hence the local price of a split $=\gamma$ I.8.

Step by step: how XGBoost actually prunes.

1. Grow the tree to max_depth (depthwise).
2. Walk bottom-up and prune nodes with $\text{Gain} < 0$ .

Why post-pruning rather than greedy stopping. A split can be bad on its own ( $\text{Gain}<0$ ) yet open the path to an excellent split below it. Greedy stopping would miss that.

Note [P]. max_depth is a ceiling, not a target depth. Trees come out asymmetric: one branch deep, another cut early. Late trees in a boosting run are usually shallower (gradients are smaller => Gain hits zero sooner).

---

[P.6] max_features / mtry - the heuristics

$$\text{classification: } m \approx \sqrt{d}, \qquad \text{regression: } m \approx \frac{d}{3}$$

Symbols. $d$ - total features, $m$ - features considered at each split.

Where from. Empirical recommendations (Breiman, ESL). This is the $\rho$ lever from I.3: smaller $m$ => more diverse trees => lower correlation => lower variance floor; but each tree is weaker (higher bias). The classic trade-off.

The XGBoost analogue - three levels that multiply:

$$m_{\text{node}} \approx d \cdot \text{colsample bytree} \cdot \text{colsample bylevel} \cdot \text{colsample bynode}$$

With all three at 0.5, a node sees roughly $0.125d$ features.

Note [P]. Tying max_features to formula I.3 is a strong move. Not "we add randomness" but "we lower $\rho$ to push down the variance floor".

---

[P.7] Bootstrap size and subsample

Bagging/RF: sampling with replacement, size $= n$ (by default). Hence 0.632 / 0.368 I.4.

Boosting (subsample): sampling without replacement, a fraction 0.5-1.0. This is stochastic gradient boosting (Friedman).

Note [P]. The with/without-replacement distinction is a detail people mix up constantly. Bagging needs replacement to get different same-size samples; boosting just takes a fraction of the rows.

---

[P.8] Feature importance: the formulas and their biases

MDI (mean decrease in impurity), a.k.a. gain in XGBoost:

$$\text{Imp}(X_j) = \frac{1}{N}\sum_{\text{trees}} \ \sum_{t : \text{ node } t \text{ splits on } X_j} \frac{n_t}{n}\Delta I(t)$$

Symbols. $\Delta I(t)$ - the impurity decrease (or Gain) at node $t$ ; $n_t/n$ - the share of samples passing through the node (coverage weighting).

Importance types in XGBoost:

Type	Formula / meaning
weight	how many times the feature was used in splits
gain	average Gain over that feature's splits
cover	average hessian sum $H$ in nodes using the feature
total_gain, total_cover	totals rather than averages

Permutation importance:

$$\text{Imp}_j = \text{err}\big(\text{data with } X_j \text{ shuffled}\big) - \text{err}(\text{original data})$$

The biases (must know).

• MDI is biased toward high-cardinality features: more candidate thresholds => more chances to stumble on a good split I.2.
• Correlated features split the credit: the tree picks one, the other gets zero.
• Permutation breaks under correlation: shuffling creates unrealistic samples outside the data's support.
• An API trap: Booster.get_score() defaults to weight, while the sklearn wrapper's feature_importances_ in modern versions defaults to gain. One model, different rankings. Always set importance_type explicitly.

The main warning. Feature importance is not a causal effect. The model exploits any association, proxy, or confounder. It's the same trap as confounding in statistical inference, wearing an importance costume.

Where else it shows up. Direct parallel: a regression coefficient on observational data is an association, not an effect.

---

[P.9] Metrics: what measures what

ROC-AUC (the interpretation that matters):

$$\text{AUC} = P\big(s(x^+) > s(x^-)\big)$$

That is the probability that a randomly chosen positive scores higher than a randomly chosen negative. Equivalent to the normalized Mann-Whitney $U$ statistic:

$$\text{AUC} = \frac{U}{n_+ n_-}$$

Symbols. $s(\cdot)$ - the model's score; $n_+, n_-$ - the number of positives and negatives.

Consequence. AUC depends only on the ordering, not the score values. Therefore monotone calibration does not change AUC. An excellent interview fact.

Brier score (calibration + discrimination):

$$\text{Brier} = \frac{1}{n}\sum_{i=1}^{n}(p_i - y_i)^2$$

Literally MSE on probabilities. A proper scoring rule.

Log-loss:

$$\text{LogLoss} = -\frac{1}{n}\sum_i \big[y_i\log p_i + (1-y_i)\log(1-p_i)\big]$$

Which when.

• you need ranking => ROC-AUC; under heavy imbalance PR-AUC is more honest;
• you need probabilities => log-loss, Brier, calibration curves.

Note [P]. Discrimination and calibration are different qualities. A model can rank perfectly (AUC = 0.95) and lie in its probabilities. If decisions are made at a monetary threshold - calibration is mandatory.

Calibration of tree models.

• Random Forest - typically under-confident (averaging pulls probabilities toward the middle).
• Boosting on log-loss - typically over-confident (pushes toward 0 and 1). Treatment: Platt scaling (a logistic layer on top of the score) or isotonic regression, on a separate set.

---

[P.10] scale_pos_weight and the base-rate shift

$$g_i \leftarrow \omega g_i, \quad h_i \leftarrow \omega h_i \quad \text{for } y_i = 1, \qquad \omega \approx \frac{n_-}{n_+}$$

where $\omega$ is scale_pos_weight.

What it does. Re-weights the positive class by multiplying its gradients and hessians.

What it breaks. Calibration. Exactly like undersampling: it shifts the base rate, so the predicted probabilities stop matching the real-world frequency.

How to choose.

• you need ranking (top-N for manual review) => scale_pos_weight is fine, calibration irrelevant;
• you need probabilities (threshold decisions in money) => better not to re-weight and tune the threshold instead; or re-weight and then calibrate.

Where else it shows up. The same story as "negative sampling breaks calibration": sampling/re-weighting moves the intercept (base rate), not the slopes. For inference (coefficients, odds ratios) it's nearly harmless; for prediction with probabilities it's critical.

---

[P.11] Histogram subtraction

$$\text{hist}(\text{parent}) = \text{hist}(\text{left}) + \text{hist}(\text{right}) \implies \text{hist}(\text{big child}) = \text{hist}(\text{parent}) - \text{hist}(\text{small child})$$

The idea. Build the histogram only for the smaller child; get the bigger one by subtraction. Nearly a 2x saving.

Where else it shows up. A standard trick in every histogram-based booster (XGBoost hist, LightGBM).

Note [P]. A good "engineering" fact - shows you've read beyond the math.

---

[P.12] Weighted quantile sketch: why the quantiles are hessian-weighted

Define the rank function for feature $k$ :

$$r_k(z) = \frac{1}{\sum_{(x,h) \in D_k} h}\sum_{(x,h)\in D_k, \ x < z} h$$

Candidate thresholds $\lbrace s_{k1}, ..., s_{kl}\rbrace$ are chosen so that

$$\big|r_k(s_{k,j}) - r_k(s_{k,j+1})\big| < \varepsilon \implies \text{roughly } 1/\varepsilon \text{ candidates}$$

Symbols. $D_k = \lbrace (x_{1k}, h_1), ..., (x_{nk}, h_n)\rbrace$ - pairs of (feature value, hessian); $\varepsilon$ - the precision (sketch_eps).

Why the weights are exactly $h_i$ . Because after Taylor the objective is equivalent to a weighted least squares with weights $h_i$ T.7. So a sample's importance for threshold selection is proportional to its hessian. The grid is built so that the total $h$ per bucket is equal, not the sample count.

Sanity check on MSE. $h_i = 1$ => weighted quantiles degenerate into ordinary ones. A special case, again.

Note [P]. "Why are the quantiles weighted?" is a marker question separating people who read the paper from people who read a blog post.

---

That was the practice tier. If one of these knobs just explained a bug you shipped last month - you know which reaction button to press ;)

New deep-dives (plus the provider discounts and free tiers I run into) land on my Telegram first: FitHappensML. Or catch me on X / Threads - ML/DS folks and app builders get a follow back.

Next up, part 3: the full derivations behind all of this, plus the cheat-sheet appendices - Part 3 - the Theory tier: full derivations and the cheat sheets