Yuuki Yamashita

Posted on Jul 12

How to Compete on Kaggle in the Generative AI Era

#ai #kaggle #datascience #machinelearning

Kaggle turned 16 this year. The leaderboard looks the same. Everything behind it has changed.

For fifteen years, the formula for winning a Kaggle competition was fairly stable: explore the data, engineer features, train models, ensemble everything, and out-grind everyone else. In 2026, one of the strongest teams on the platform won a competition by orchestrating three LLM agents that wrote 600,000 lines of code and ran 850 experiments — while more than half of all winning teams were still individuals, and nine winners trained entirely on free Kaggle Notebooks.

Both of those facts are true at the same time. That tension is exactly what this article is about: what Kaggle was, what it has become, and how to actually compete on it today.

Part 1: A Short History of Kaggle

2010–2015: The tabular years

Kaggle was founded in April 2010 by Anthony Goldbloom and Ben Hamner in Melbourne, Australia, as a marketplace connecting companies that had prediction problems with a global pool of data scientists who wanted to solve them. Jeremy Howard joined as an early user and later became President and Chief Scientist; in 2011 the company raised $12.5 million.

The early competitions were overwhelmingly tabular: predict insurance claims, credit defaults, retail demand. The winning toolkit was feature engineering plus gradient boosting — random forests early on, then XGBoost, which effectively became famous through Kaggle after dominating competitions like the 2014 Higgs Boson Machine Learning Challenge. If you competed in this era, your edge was domain intuition, creative features, and rigorous cross-validation.

2016–2019: The deep learning wave

After the ImageNet moment, computer vision and NLP competitions took over. Convolutional networks, transfer learning, and pretrained backbones became the default; competitions shifted from "who has the best features" to "who can fine-tune and ensemble deep models most effectively." Kaggle launched Kernels (later Notebooks) with free GPUs, which lowered the hardware barrier dramatically.

Two milestones bookend this era: Google acquired Kaggle in March 2017 (announced by Fei-Fei Li at Google Cloud Next), and the platform crossed one million registered users the same year.

2020–2023: Transformers everywhere, and the rise of code competitions

BERT-style encoders swallowed the NLP track. Vision Transformers started challenging CNNs. Just as importantly, Kaggle changed the format of competing: code competitions — where you submit a notebook that runs offline, with strict time limits and no internet access — became the norm. This one design decision still shapes strategy today: you cannot simply call a frontier model API from inside most competitions. Whatever intelligence you use at inference time has to fit inside the compute box Kaggle gives you.

2024–2026: The generative AI era

This is where the platform itself transformed. Three things happened at once:

Competitions about LLMs. Challenges like the AI Mathematical Olympiad (AIMO) progress prizes — with a grand prize north of $1.5 million — and the ARC Prize (aimed at abstract reasoning, with $2M in prizes for 2026) made "build a reasoning system" the headline competition genre. ARC-AGI-3, launched in March 2026, goes further: you build agents that play interactive games, testing exploration, planning, and memory rather than static prediction.
Competitions judged between models. Kaggle Game Arena, launched with Google DeepMind in August 2025, pits frontier models against each other in chess — and, since early 2026, poker and Werewolf — in an all-play-all format. Kaggle is no longer only a place where humans compete on data; it's becoming a public benchmarking venue where the models are the competitors.
LLMs as the competitor's power tool. The most visible example: an NVIDIA team won a 2026 churn-prediction competition by directing three LLM agents (they cite GPT-5.4 Pro, Gemini 3.1 Pro, and Claude Opus 4.6) through the classic Grandmaster workflow — EDA, baselines, feature engineering, hill climbing, stacking — generating 600K+ lines of code and 850 experiments, culminating in a four-level stack of 150 models.

Along the way, the platform grew from 15 million users in 2023 to roughly 29 million in 2025, hosting the largest prize pool of any competition platform (~$3.7M across 2025, out of $16M+ industry-wide).

Part 2: How to Actually Compete in 2026

Here is the surprising part: the fundamentals did not get replaced. They got compressed.

1. The Grandmaster playbook still wins — LLMs just execute it faster

Look at what the agent-powered winners actually did: EDA → baseline → feature engineering → hill climbing → stacking. That is the same playbook Grandmasters have taught for a decade. The LLMs didn't invent a new strategy; they removed the typing.

Practical takeaway: your value has moved up one level of abstraction. Instead of writing every experiment yourself, you design the experiment tree, and let AI assistants generate and run the branches. The competitor who knows which 50 experiments matter beats the one who blindly runs 500.

2. Validation design is now your single biggest edge

When code generation is nearly free, everyone can produce hundreds of models. What LLMs still reliably get wrong: leakage, unstable cross-validation schemes, and mismatches between local CV and the leaderboard. The 2025 winners' write-ups repeat the same theme — humans caught the leakage, chose the CV split, and decided what "trust your CV" meant for that particular competition. If you invest your scarce human attention anywhere, invest it here.

3. Know what still wins by modality

Based on winning solutions across 2025:

Tabular: Gradient-boosted trees remain king — XGBoost and LightGBM (14 winning solutions each), CatBoost close behind. AutoGluon is quietly strong; one winner reported it beat a hand-tuned XGBoost/LightGBM ensemble using a fraction of the compute.
Computer vision: 2025 was the first year Transformer architectures outnumbered CNNs among winners — DINOv2 and Swin for classification, YOLOv8/v11 for detection.
NLP: Decoder-only models dominate, and open-weight Qwen models appeared in nearly every winning text solution. BERT-style encoders have almost disappeared from the winners' circle.
Frameworks: PyTorch appeared in 44 winning deep learning solutions; TensorFlow in one.

4. Fine-tune open weights; don't plan around API calls

Because code competitions run offline, the generative AI era on Kaggle is really the open-weights era. Winning LLM-competition solutions fine-tune and quantize models like Qwen to fit inside Kaggle's GPU/time budget, and use tricks like synthetic data generation and speculative decoding (both cited in the $250K+ AIMO Progress Prize 2 win by the NemoSkills team). Skills that pay off: LoRA/QLoRA fine-tuning, quantization, inference optimization under a deadline.

5. Compute matters less than you'd fear

Yes, H100s overtook A100s among winners in 2025, and one team used 512 H100s for a single training run (~$60K). But in the same year, nine winning solutions were trained entirely on free Kaggle Notebooks, over half of winning teams were solo, and more than half were first-time winners. The playing field is more open than the headlines suggest — largely because the offline-notebook format caps how much raw compute can matter at inference time.

6. Watch the new genre: agent competitions

Simulation competitions (Lux AI, Halite lineage), Game Arena, and ARC-AGI-3 point at where the platform is heading: you submit an agent, not a prediction file. The skills are different — environment design, planning under uncertainty, evaluation harnesses — and the field is much less crowded than tabular or CV. If you're starting on Kaggle in 2026, this is the least-saturated place to build a reputation.

7. Use LLMs where they're strong, verify where they're weak

A working division of labor, distilled from recent winners' reports:

Delegate to AI assistants	Keep human
Boilerplate, pipelines, refactoring	Problem framing, metric analysis
EDA code and first-pass insights	Spotting leakage and CV design
Hyperparameter sweep scaffolding	Deciding which ideas are worth compute
Survey of prior solutions	Reading the competition's "meta" and rules
Ensembling mechanics	Final-submission risk management

The failure modes are well documented: hallucinated code paths, misread task context, subtle bugs in generated pipelines. Winners treat LLM output as a fast junior teammate's draft — reviewed, tested, never trusted blindly.

Part 3: What Kaggle Is Becoming

Step back and the trajectory is clear. Kaggle started as a labor marketplace (2010), became a learning platform with free compute (2016+), and is now evolving into something like a public evaluation layer for AI itself — hosting benchmarks (Game Arena, Kaggle Benchmarks), agent competitions (ARC-AGI-3), and human-AI hybrid contests simultaneously.

For competitors, that means the question "will LLMs make Kaggle pointless?" has been answered in practice: no — but they changed what is scarce. Code is no longer scarce. Experiments are no longer scarce. What's scarce is judgment: validation design, knowing the playbook well enough to direct agents through it, and the taste to tell a promising direction from an expensive dead end.

Which, ironically, is exactly what Kaggle has always taught. The leaderboard still doesn't care how the code got written.

If you're starting today: pick one active competition, build a baseline by hand so you understand the data, then let AI assistants accelerate everything after that. The playbook is public. The judgment is yours to earn.

DEV Community