DEV Community

Malcolm Low
Malcolm Low

Posted on Originally published at malcolmlow.com

Pocket Data Science: Exploring Regression Baselines and Ensembles on Android with Antigravity CLI

#machinelearning #python #android #datascience

A hands-on experiment exploring regression baselines, metric properties in log space, Ridge regression, and CatBoost on Kaggle's House Prices dataset using an Android phone running Termux and Google Antigravity CLI.

In our previous articles in the Pocket Data Science series (Titanic and Spaceship Titanic), we explored binary classification on mobile hardware using Google Antigravity CLI (agy) running inside an Android Linux environment.

In this follow-up, we turn to tabular regression with another classic entry-level benchmark: Kaggle's House Prices: Advanced Regression Techniques (based on the Ames, Iowa housing dataset compiled by Dean De Cock).

Regression brings a distinct set of practical considerations compared to binary classification: target skewness, multicollinear continuous variables, and evaluation metrics with non-linear properties. Instead of jumping straight into complex models, we walked through a step-by-step progression:

  1. Measuring the variance of an empirical random baseline.
  2. Deriving the mathematically optimal single-number constant under Root Mean Squared Logarithmic Error (RMSLE).
  3. Using simple piecewise constant lookups to see how far basic feature grouping can go.
  4. Applying standard regularized linear regression (Ridge) and gradient boosted trees (CatBoost).
  5. Combining both models with a basic geometric blend.

All steps were executed locally on an ARM64 Android device using Google Antigravity CLI (agy) inside a Termux PRoot environment. Across eight iterations, the predictions moved from a naive random score of 0.56940 to an ensembled score of 0.12363 (Rank #704 of 3,734, Top 18.8%) on the public leaderboard.

Here is an objective breakdown of what each step contributed and what the results tell us about tabular regression mechanics.

1 · The Local Mobile Setup

The entire pipeline was run within a standard Linux userland hosted on an Android phone:

  • Environment: Android 14 running Termux with a Debian userspace via PRoot Distro on 64-bit ARM (aarch64).
  • Assistant / CLI: Google Antigravity CLI (agy) used as an interactive terminal pair programmer to run commands, inspect logs, and manage submissions.
  • Libraries: Python 3.14 with pandas, numpy, scikit-learn, catboost, and the official kaggle CLI.
  • Resources: Execution was kept lightweight (under 2GB peak RAM usage) with 5-fold cross-validation running comfortably in a few seconds to a couple of minutes per model.

2 · Summary of Experiment Progression

Below is the chronological sequence of submissions evaluated against Kaggle's public test set:

Exp Strategy Description Kaggle RMSLE LB Rank Percentile Teams Ahead / Behind
01 Empirical Random Sample with replacement from train SalePrice 0.56940 #3,586 Bottom 4% 148 beaten
03 Constant Mean Single scalar: $180,921.20 0.42577 #3,535 Bottom 5% 199 beaten
02 Constant Median Single scalar: $163,000.00 0.41657 #3,528 Bottom 5% 206 beaten
04 Geometric Mean Single scalar: $166,716.73 ($\mathbb{E}[\ln Y]$ in log space) 0.41637 #3,527 Bottom 5% 207 beaten
05a 1D Piecewise Constant 10 values: log-mean by OverallQual 0.22613 #3,355 Top 90% 379 beaten
05b 2D Piecewise Constant ~170 values: log-mean by Neighborhood $\times$ OverallQual 0.20945 #3,320 Top 89% 414 beaten
06 Ridge Regression 5-Fold CV, Median Imputer, Scaler, One-Hot Encoding 0.13002 #1,438 Top 38.5% 2,296 beaten
07 CatBoost Regressor 5-Fold CV, Native Categoricals, 1,500 trees 0.12601 #992 Top 26.6% 2,742 beaten
08 Blended Ensemble 80% CatBoost + 20% Ridge in log space 0.12363 #704 Top 18.8% 3,030 beaten

(Note: Leaderboard statistics based on 3,734 total active teams on Kaggle as of September 2026).

3 · The Baseline Hierarchy: Why Constants Reduce Error

Before training parameterized models, establishing a baseline hierarchy provides clear lower bounds on performance.

Exp 01: The Empirical Random Baseline (0.56940)

In Experiment 01, we drew random samples from the training set's SalePrice distribution ($34,900 to $755,000) for each test row.

Because predictions were randomly assigned without regard to house features, prediction variance was high. A modest home could easily receive a $600,000 prediction, while a large home could receive $45,000. On the public leaderboard, this scored 0.56940 (Rank #3,586).

Why Constant Predictions Reduce Error Immediately

In the classical bias-variance decomposition:
$$\text{Expected Loss} = \text{Bias}^2 + \text{Variance} + \text{Irreducible Noise}$$

A single constant prediction across the entire dataset has zero variance by definition ($\text{Var}(\hat{y}) = 0$). Even though the bias is substantial, eliminating prediction variance drops the error from 0.569 to around 0.416–0.425 (an immediate reduction of over 25%).

4 · Metric Alignment: Mean vs. Median vs. Geometric Mean

Kaggle evaluates House Prices on Root Mean Squared Logarithmic Error (RMSLE):

$$\text{RMSLE} = \sqrt{\frac{1}{N} \sum_{i=1}^{N} \left(\ln(1 + \hat{y}_i) - \ln(1 + y_i)\right)^2}$$

This metric changes which constant is optimal:

  1. Arithmetic Mean ($180,921.20 — Score: 0.42577):

    The sample mean minimizes squared errors in raw dollars ($\sum (y_i - C)^2$). However, because Ames home prices are right-skewed by a small number of expensive homes (up to $755,000), the mean is pulled upward away from the typical home. In log space, overpredicting typical homes incurs an asymmetric penalty.

  2. Median ($163,000.00 — Score: 0.41657):

    The median minimizes Mean Absolute Error (MAE) in dollars. Because it is robust to extreme outliers, it sits closer to the center of mass of the distribution, outperforming the arithmetic mean by 0.0092 RMSLE (+7 leaderboard spots).

  3. Geometric Mean ($166,716.73 — Score: 0.41637):

    Under RMSLE, the loss is standard mean squared error on $z_i = \ln(1 + y_i)$. Setting $\frac{\partial}{\partial \hat{z}} \sum (\hat{z} - z_i)^2 = 0$ yields:
    $$\hat{z}^* = \frac{1}{N} \sum_{i=1}^{N} z_i \implies C^* = \exp(\overline{\ln(1+y)}) - 1 \approx \mathbf{\$166,716.73}$$
    This is the theoretical minimizer in log space for a single scalar. On Kaggle's test set, it yielded 0.41637, slightly edging out the median.

5 · The Conceptual Bridge: Piecewise Constant Models

Before reaching for standard scikit-learn models, we tested a simple non-parametric idea: Piecewise Constant Lookup Tables.

In statistical theory, regression trees (CART) are essentially algorithms that automatically partition input space into rectangular piecewise constant regions. We can test this manually by grouping test instances by one or two key features:

# 1D Piecewise Constant: Grouping by Overall Quality (1 to 10)
train['log_y'] = np.log1p(train['SalePrice'])
qual_lookup = train.groupby('OverallQual')['log_y'].mean()

pred_log = test['OverallQual'].map(qual_lookup)
test['SalePrice'] = np.expm1(pred_log)
Enter fullscreen mode Exit fullscreen mode
OverallQual Tier Train Count Median Price Geometric Mean
1 2 $50,150 $48,962
3 20 $86,250 $83,908
5 397 $133,000 $130,700
7 319 $200,141 $203,165
9 43 $345,000 $359,787
10 18 $432,390 $408,933
  • Exp 05a (1D Piecewise on OverallQual): Using just 10 discrete values dropped the score from 0.416 to 0.22613 (Rank #3,355).
  • Exp 05b (2D Piecewise on Neighborhood $\times$ OverallQual): Accounting for geographic area plus quality dropped the score further to 0.20945 (Rank #3,320), placing ahead of 414 submissions without running any optimization algorithm.

6 · Machine Learning Baselines: Ridge Regression and CatBoost

With baselines established, we trained two standard tabular models using 5-fold cross-validation. For both models, the target variable was transformed via $\ln(1 + y)$ so standard squared-error loss aligns directly with RMSLE.

Model 1: Regularized Ridge Regression (Exp 06 — 0.13002)

Real estate data contains significant collinearity (e.g., GarageCars vs GarageArea, TotalBsmtSF vs 1stFlrSF). Ordinary Least Squares (OLS) can produce erratic coefficient swings under collinear features.

  • Pipeline: Numeric features were median-imputed and standardized (StandardScaler). Categorical features were imputed with a "Missing" token and one-hot encoded.
  • Model: RidgeCV tested 50 regularization penalties ($\alpha \in [10^{-2}, 10^3]$), converging around $\alpha \approx 18–23$.
  • Results: 5-fold Out-Of-Fold (OOF) RMSLE was 0.15125, scoring 0.13002 on the public test set (Rank #1,438, Top 38.5%).

Model 2: CatBoost Regressor (Exp 07 — 0.12601)

The Ames dataset contains 44 categorical columns with varying cardinality. CatBoost natively computes ordered target statistics on categoricals without requiring expansive one-hot encodings.

  • Setup: 1,500 trees, learning rate of 0.03, depth of 6, and L2 leaf regularization of 3.0, evaluated across 5 folds with early stopping.
  • Results: 5-fold OOF RMSLE was 0.12803, scoring 0.12601 on the public test set (Rank #992, Top 26.6%).

7 · Blending the Models (Exp 08 — 0.12363)

Because linear models and decision trees make structurally different assumptions, ensembling them often reduces residual variance:

  • Ridge models smooth global trends across continuous square footage and age.
  • CatBoost captures localized non-linear thresholds and categorical interactions.

Because the evaluation metric operates in log space, predictions should be combined via a geometric blend:

$$\ln(\hat{y}{\text{blend}}) = w \cdot \ln(\hat{y}{\text{Ridge}}) + (1 - w) \cdot \ln(\hat{y}_{\text{CatBoost}})$$

$$\hat{y}{\text{blend}} = \exp\left(0.20 \cdot \ln(1 + \hat{y}{\text{Ridge}}) + 0.80 \cdot \ln(1 + \hat{y}_{\text{CatBoost}})\right) - 1$$

  • Final Score: 0.12363
  • Leaderboard Rank: #704 of 3,734 competitors (Top 18.85%)
  • Net Improvement over CatBoost alone: -0.00238 RMSLE (+288 leaderboard places).

8 · Practical Takeaways & Limitations

  1. Understand Metric Mechanics First: If an evaluation metric uses log-transformed targets, evaluating constant baselines and ensembling in log space is mathematically required. Optimizing in raw dollars shifts model attention toward high-priced outliers.
  2. Contextualizing Leaderboard Ranks: While reaching the top 19% (0.12363) with simple scripts is encouraging, it is important to recognize that introductory Kaggle competitions feature many inactive or incomplete submissions. In a competitive setting, moving deeper into the top 10% (< 0.121) typically requires detailed neighborhood clustering, outlier pruning (such as Frank Harrell's known >4,000 sq ft Ames outliers), and stacking multiple diverse architectures.
  3. The Value of a Baseline Staircase: Moving methodically from Random (0.569) $\to$ Constant (0.416) $\to$ Piecewise Table (0.209) $\to$ Linear Model (0.130) $\to$ GBDT (0.126) $\to$ Blend (0.123) makes it easy to audit exactly where performance gains originate.
  4. Feasibility on Mobile Userspace: Consumer mobile hardware running Termux and PRoot can comfortably execute 5-fold cross-validation and tabular pipelines using modern tools like CatBoost and scikit-learn without specialized cloud instances.

Originally published on malcolmlow.com.

Top comments (0)