Eliminating Accidental Submissions in Kaggle Work: The Kaggle CLI, a Confirmation Wrapper, and Measuring on Production Hardware

#kaggle #python #cli #automation

This article was originally published in Japanese on Zenn.

Introduction

I've been working on Kaggle with help from LLM agents and scripts. Automation makes things easier, but it also nearly caused accidents, like the workflow going all the way to submission without anyone checking.

In this article, I'll cover how to work with notebooks using the Kaggle CLI, a wrapper that always puts a human confirmation step in the loop, and what I learned from the differences between my validation environment and the production environment.

1. Checking notebook status with the Kaggle CLI

Setting up the token

With the newer token method, you put your token in the KAGGLE_API_TOKEN environment variable.

export KAGGLE_API_TOKEN="(your token issued by Kaggle)"

Getting notebook status and output

You can check the status of a notebook running on Kaggle and fetch its output from the CLI.

# Check whether it's running, finished, or failed
kaggle kernels status <username>/<notebook-name>

# Download the output files locally
kaggle kernels output <username>/<notebook-name>

Since you can see the status without opening a browser, it's much easier to handle from scripts and agents.

2. A confirmation wrapper to prevent accidental submissions

I once had an automated batch go all the way to submission without any confirmation. Submissions are often limited in number, and they can't be undone.

So I built a wrapper that splits the path to submission into these steps:

Show which file will be submitted
Confirm with y/N (pressing just Enter means "No")
Run it
Validate the output (whether the file exists, whether the format is correct, etc.)
Only display the submit command and stop there

For the final submission, a human runs the displayed command themselves. This way, even if something breaks somewhere in the automation, a human always takes a look right before submission.

The confirmation part can be written in Python in just a few lines, for example:

def confirm(message: str) -> bool:
    answer = input(f"{message} [y/N]: ").strip().lower()
    return answer == "y"

if not confirm("Run with this file?"):
    raise SystemExit("Aborted")

The key is making "No" the default. Even when it's called from an agent or script with no input, it won't proceed on its own.

3. Measure time-limited workloads on the same hardware as production

When I was doing local validation on a fast GPU, the trends in the results sometimes differed from those on the slower GPU used in production.

When a workload has a time limit, slower hardware alone can cause it to be cut off partway, or change what you can get done within the limit. Just because something "fits in time" on a fast environment doesn't mean it will in production.

I learned that for time-limited workloads, it's important to measure on the same hardware as production.

Summary

Once you set KAGGLE_API_TOKEN, the Kaggle CLI lets you track your notebooks with kernels status and kernels output, no browser needed
Even with automation, always put a human confirmation step right before submission (default to "No", and only display the submit command)
Measure time-limited workloads on the same hardware as production

The more I automate, the more I feel it's important to decide up front "where a human steps in to stop things."

DEV Community