Kaggle is making AI benchmark creation effortless

Nicholas Kang (Nick) — Thu, 04 Jun 2026 15:51:23 +0000

As AI models evolve from simple chatbots into reasoning agents that write code, use tools and solve complex problems, traditional benchmarks are no longer enough. The community needs dynamic, rigorous evaluations — built by the people who use these models in the real-world.

That’s why we launched Kaggle Benchmarks. Since then, the global AI community has created more than 10,000 evaluation tasks, creating the trustworthy, transparent public leaderboards that help labs measure and accelerate AI progress.

Today, we are taking the next step by launching local development for Kaggle Benchmarks.

Use Kaggle Benchmarks from your local development environment

Until now, creating evaluation tasks meant working exclusively in Kaggle's web-based notebook editor, instead of developers’ preferred stack to build with.

Our new update enables developers to create, validate, push, run and download tasks directly from their local development environments like Antigravity, VSCode, Cursor and coding agents. This update is designed to meet developers where they work, making the journey from idea to evaluation faster and more intuitive.

Build evaluation tasks in natural language with AI coding agents

Local development also unlocks a powerful new workflow: using AI coding agents to write benchmark tasks through the write-kaggle-benchmarks skill. This skill comprises a set of structured instructions that teaches a coding agent how to build tasks using the kaggle-benchmarks SDK and the Kaggle CLI.

To add this skill to your agent, simply ask your agent to:

“Install the write-kaggle-benchmarks skill: https://github.com/Kaggle/kaggle-skills”

Once installed, you can describe an evaluation in plain language and get a working task on Kaggle. For example, you can tell your agent:

Using the write-kaggle-benchmarks skill, build a task that asks the model if "300+140=460 is correct?"

These powerful capabilities are driven by the new commands that we have built for Benchmarks in the Kaggle CLI.

Understand why community-driven evaluations matter

We built Kaggle Benchmarks to democratize trustworthy AI evaluations. We believe that if a capability can be measured, labs will race to improve it. By providing these clear, objective signals, our hope is to empower AI labs to drive model improvements in the areas that matter most.

For AI to truly benefit humanity, evaluations must reflect the full diversity of real-world challenges. We believe this launch is a significant step toward enabling anyone, anywhere, to build the evaluations that will shape the future of AI.

Ready to build? Try Kaggle Benchmarks today.

Additional resources

Read the docs for kaggle-cli on GitHub
Install the write-kaggle-benchmark skill
🎁 Idea → eval? Show us. Post your task + workflow by tagging @kaggle on X or LinkedIn by July 1st for a chance to win Kaggle swag and a social shoutout
Watch the product demo on YouTube

Introducing Community Benchmarks on Kaggle

Nicholas Kang (Nick) — Wed, 14 Jan 2026 20:54:10 +0000

Today, Kaggle is launching Community Benchmarks, which lets the global AI community design, run and share their own custom benchmarks for evaluating AI models. This is the next step after we launched Kaggle Benchmarks last year, to provide trustworthy and transparent access to evaluations from top-tier research groups like Meta’s MultiLoKo and Google’s FACTS suite.

Why community-driven evaluation matters

AI capabilities have evolved so rapidly that it’s become difficult to evaluate model performance. Not long ago, a single accuracy score on a static dataset was enough to determine model quality. But today, as LLMs evolve into reasoning agents that collaborate, write code and use tools, those static metrics and simple evaluations are no longer sufficient.

Kaggle Community Benchmarks provide developers with a transparent way to validate their specific use cases and bridge the gap between experimental code and production-ready applications.

These real-world use cases demand a more flexible and transparent evaluation framework. Kaggle’s Community Benchmarks provide a more dynamic, rigorous and continuously evolving approach to AI model evaluation — one shaped by the users building and deploying these systems everyday.

How to build your own benchmarks on Kaggle

Benchmarks start with building tasks, which can range from evaluating multi-step reasoning and code generation to testing tool use or image recognition. Once you have tasks, you can add them to a benchmark to evaluate and rank selected models by how they perform across the tasks in the benchmark.

Here’s how you can get started:

Create a task: Tasks test an AI model’s performance on a specific problem. They allow you to run reproducible tests across different models to compare their accuracy and capabilities.
Create a benchmark: Once you have created one or more tasks, you can group them into a Benchmark. A benchmark allows you to run tasks across a suite of leading AI models and generate a leaderboard to track and compare their performance.

Once you build your benchmark, here’s what benefits you’ll see:

Broad model access: Free access (within quota limits) to state-of-the-art models from labs like Google, Anthropic, DeepSeek and more.
Reproducibility: Benchmarks capture exact outputs and model interactions so results can be audited and verified.
Complex interactions: They support testing for multi-modal inputs, code execution, tool use and multi-turn conversations.
Rapid prototyping: They allow you to quickly design and iterate on creative new tasks.

These powerful capabilities are powered by the new kaggle-benchmarks SDK. Here are a few resources for getting started:

Benchmarks Cookbook: A guide to advanced features and use cases.
Example tasks: Get inspired with a variety of pre-built tasks.
Getting started: How to create your first task & benchmark

How we’re shaping the future of AI evaluation

The future of AI progress depends on how models are evaluated. With Kaggle Community Benchmarks, Kagglers are no longer just testing models, they’re helping shape the next generation of intelligence.

Ready to build? Try Community Benchmarks today.

DEV Community: Nicholas Kang (Nick)