Hyperparameter Tuning with GASearchCV on a Real Dataset

If you've trained more than a handful of ML models, you've run into the same wall: GridSearchCV is thorough but painfully slow once your parameter space grows, and RandomizedSearchCV is fast but blind — it has no idea which region of the search space is actually promising.

While working on a student performance prediction model (forecasting pass/fail outcomes from academic and behavioral features), I needed to tune a Random Forest classifier without burning hours on an exhaustive grid search. That's what pushed me to try sklearn-genetic-opt — specifically its GASearchCV class, which replaces brute-force search with an evolutionary algorithm.

This post walks through what GASearchCV actually does differently, and shows a real before/after comparison against GridSearchCV on my dataset.

The problem with the usual approach

• GridSearchCV evaluates every single combination in your parameter grid. Add two more values to one parameter and your runtime multiplies.
• RandomizedSearchCV samples randomly, so it's faster, but it doesn't learn from earlier results — a combination that scored well tells the search nothing about where to look next.

GASearchCV treats each hyperparameter combination as an "individual" in a population. Individuals that score well are more likely to pass their parameters into the next generation, with some random mutation mixed in to keep exploring. Over a few generations, the search converges toward strong regions of the hyperparameter space instead of wasting time on the whole grid.

Setup

pip install sklearn-genetic-opt

import pandas as pd
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score
from sklearn_genetic import GASearchCV
from sklearn_genetic.space import Integer, Categorical

# Load dataset — UCI Student Performance dataset (semicolon-separated)
df = pd.read_csv("student_data.csv", sep=';')

# Build a binary pass/fail target from the final grade (G3, scale 0-20)
# 10+ is the standard pass threshold for this dataset
df["pass"] = (df["G3"] >= 10).astype(int)

# Drop the raw grade columns so the model predicts from behavioral/academic
# features rather than just reading off the final score directly
X = df.drop(columns=["G1", "G2", "G3", "pass"])
y = df["pass"]

# One-hot encode categorical columns (school, sex, address, Mjob, Fjob, etc.)
X = pd.get_dummies(X, drop_first=True)

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

A quick note on G1/G2: they're first and second period grades, and they correlate very strongly with G3 — keeping them in would let the model mostly just read off the answer instead of learning from behavioral and academic context (study time, failures, absences, family background, etc.). Dropping them makes for a harder, more honest "early prediction" problem, which is the more interesting one to tune hyperparameters for anyway.

Baseline: GridSearchCV

param_grid = {
    "n_estimators": [50, 100, 150, 200],
    "max_depth": [3, 5, 7, 10, None],
    "min_samples_split": [2, 4, 6, 8],
    "criterion": ["gini", "entropy"],
}

grid_search = GridSearchCV(
    estimator=RandomForestClassifier(random_state=42),
    param_grid=param_grid,
    cv=5,
    scoring="accuracy",
    n_jobs=-1,
)

grid_search.fit(X_train, y_train)
print("Grid best score:", grid_search.best_score_)
print("Grid best params:", grid_search.best_params_)

This grid has 4 × 5 × 4 × 2 = 160 combinations, each evaluated across 5 CV folds — 800 model fits total.

Same search, with GASearchCV

param_grid_ga = {
    "n_estimators": Integer(50, 200),
    "max_depth": Integer(3, 10),
    "min_samples_split": Integer(2, 8),
    "criterion": Categorical(["gini", "entropy"]),
}

ga_search = GASearchCV(
    estimator=RandomForestClassifier(random_state=42),
    cv=5,
    scoring="accuracy",
    param_grid=param_grid_ga,
    population_size=15,
    generations=10,
    n_jobs=-1,
    verbose=True,
)

ga_search.fit(X_train, y_train)
print("GA best score:", ga_search.best_score_)
print("GA best params:", ga_search.best_params_)

Instead of defining discrete lists, you define ranges (Integer, Categorical, Continuous) — the algorithm searches within them rather than exhaustively enumerating every point. With a population of 15 and 10 generations, this evaluates roughly 150 combinations total — fewer fits than the full grid, but concentrated on the regions that keep performing well across generations, rather than spread evenly (and wastefully) across the whole space.

Comparing results

print("Grid test accuracy:", accuracy_score(y_test, grid_search.predict(X_test)))
print("GA test accuracy:", accuracy_score(y_test, ga_search.predict(X_test)))

Here's what I actually got on the student performance dataset:

	Best CV score	Test accuracy	Best params found
GridSearchCV	0.7375	0.6835	n_estimators=150, max_depth=None, min_samples_split=8, criterion='entropy'
GASearchCV	0.7469	0.6582	n_estimators=80, max_depth=8, min_samples_split=5, criterion='entropy'

Interesting split: GASearchCV found a slightly better cross-validation score than the full grid search, but ended up marginally lower on the held-out test set. With only ~79 rows in the test split, that gap is well within noise — not a meaningful win either way. What I found more useful than the accuracy numbers themselves was how each search got there. GridSearchCV had to check all 160 combinations from the fixed grid (800 model fits across 5 folds) to land on its answer. GASearchCV, working over continuous ranges rather than a fixed list, converged on a strong region in about 10 generations — you can see this directly in the verbose log:

gen  evals    avg      best     div    unique   stag   mut   sel   events
0    15       0.71859  0.73740  0.500  1.000    0      -     -     -
1    30       0.72177  0.73740  0.411  1.000    1      0.100 3     dup=12
2    30       0.72576  0.74687  0.321  0.800    0      0.100 3     dup=10
...
10   30       0.73194  0.74687  0.286  0.800    8      0.200 3     div,imm=3,dup=10

The best column climbs from 0.737 to 0.747 and then plateaus — you can literally watch the population converge, generation by generation, which is something a flat grid search can't show you at all.

Also worth noting: GASearchCV also landed on a smaller model (n_estimators=80 vs. 150, max_depth=8 vs. unbounded) with essentially the same performance — a useful side effect if inference speed or model size matters to you, not just accuracy.

sklearn-genetic-opt also ships built-in visualization helpers, which I found genuinely useful for understanding why the search converged where it did:

from sklearn_genetic.plots import plot_fitness_evolution, plot_search_space

plot_fitness_evolution(ga_search)
plot_search_space(ga_search)

The best-fitness curve jumps from 0.7374 to 0.7469 right at generation 2, then flatlines for the remaining 8 generations — a clean, visible signal that the population converged early and stayed there. This is exactly the kind of detail a plain grid search can't show you: you only get the final number, not the story of how the search got there.

This one turned out to be more useful than I expected. Looking at the max_depth vs. score panel, sampled scores cluster noticeably higher around max_depth 7–8 — which lines up exactly with the best params GA landed on (max_depth=8). Same pattern shows up for min_samples_split: scores trend higher in the 4–6 range, and the best run used 5. In other words, the plot doesn't just show that the search picked those values — it shows why, visually, which made it much easier to trust the result instead of treating it as a black box.

plot_fitness_evolution shows the best and average fitness score per generation — you can visually confirm the population is actually improving over time rather than randomly wandering. plot_search_space shows the distribution of sampled hyperparameter values, which makes it obvious which regions the algorithm favored.

When genetic search is worth it — and when it isn't

Use GASearchCV when:

• Your parameter grid is large enough that a full grid search is impractically slow
• You're working with continuous ranges rather than a handful of discrete values
• You want visual insight into how the search converged, not just the final answer

Stick with GridSearchCV when:

• Your search space is small (a handful of combinations) — the overhead of an evolutionary search isn't worth it
• You need a guaranteed exhaustive answer for a small, well-understood grid, e.g. for a reproducible benchmark

Takeaway

On my student performance model, GASearchCV didn't clearly outperform GridSearchCV on test accuracy — the two landed within noise of each other, which is a realistic outcome, not every tuning method is going to beat every other one every time. What it did give me was a smaller, comparably-performing model, and — more importantly — visibility into how the search converged, generation by generation, instead of just a final answer with no story behind it. That's useful on its own: when you're presenting results, being able to show a fitness curve climbing and plateauing is a stronger explanation than "I tried 160 combinations and this one won."

If your hyperparameter space is large or continuous, it's worth running alongside your usual grid/random search — not necessarily as a replacement, but as another lens on the same problem.

Repo: rodrigo-arenas/Sklearn-genetic-opt

---

Written by Saadgi Puniwala — B.Tech CSE (Data Science), Dayananda Sagar University. Find more of my work on my portfolio.