DEV Community: kohei matsunobu

What Happens When You Tell an LLM a Story Backward?

kohei matsunobu — Sat, 07 Nov 2026 10:58:12 +0000

Fluent answers are not the whole story

When I build with an LLM, I usually start with practical questions: Is the answer useful? Can someone check its sources? Does it make sense?

That last question hides several different problems. An answer can be grammatically correct and still put events in a strange order. How could we measure that distinction without simply asking another model to judge it?

I'm Kohei Matsunobe, a second-year correspondence student at ZEN University. I build KamiMeg, an iOS goshuin (shrine-stamp) app with a generative-AI shrine guide, and a Japanese PDF RAG demo that makes its sources visible. A recent paper by Takahiro Sagawa, a professor in the Department of Applied Physics, Graduate School of Engineering at the University of Tokyo, caught my attention because it approaches this question from an unexpected direction: stochastic thermodynamics.

This is a builder's reading note based on the university's press release and the paper's abstract, not a reproduction of its experiments.

Change the order, not the words

Consider this example, translated from the press release:

The cup fell on the floor. The cup broke.

Now reverse the sentences:

The cup broke. The cup fell on the floor.

Both versions contain readable sentences. But the first follows a familiar progression; the second makes me look for a different explanation. It is not impossible, just different.

The research compares how likely such sequences are under a language model. Crucially, it reverses sentence order while preserving the word order inside each sentence.

Reversing every token would break grammar. The resulting probability difference could mostly reflect broken syntax rather than anything interesting about events. Sentence-level reversal is a way to change the relationship between sentences without destroying their internal structure.

That experimental choice is what first made the work click for me: a useful measurement starts by deciding what to disturb and what to leave intact.

Thermodynamics without measuring GPU heat

The quantity at the center of the paper is entropy production: a way to quantify irreversibility by comparing the likelihood of forward and reversed sequences.

Here, thermodynamics does not mean measuring the computer's physical heat output or energy consumption. The object of study is the model's generation process and its directional asymmetry.

The theory is broader than text. It provides a unified framework for autoregressive generative models, ranging from Transformers to Kalman filters.

There is a technical obstacle: the next output can depend on a long history, not just the immediately previous output. That makes the observed process non-Markovian. Trying to enumerate every possible history would quickly become impractical.

Sagawa's framework shows how to estimate entropy production efficiently without enumerating all those histories. It also decomposes entropy production into non-negative contributions at each timestep, using an information-theoretic perspective: inferring the past from the future.

I find that reversal of viewpoint appealing. We normally discuss these models as predictors of what comes next; this theory also asks what looking backward can reveal.

What the language-model experiments suggest

The experiments used GPT-2 (124M) and Qwen3-4B-Base. Texts describing sequential events tended to have larger sentence-level entropy production than texts listing independent facts. In other words, reversed generation was relatively less likely for the sequential-event texts.

That suggests the models' generation probabilities reflect something about event order. It offers a quantitative probe for investigating what is often called a model's internal “world model.”

It does not establish that an LLM understands causality in the same way a person does.

There is also an important dataset caveat: the evaluation texts were generated by prompting commercial generative AI. Their construction could introduce features that matter to the comparison. The press release explicitly calls for further verification, including text sets constructed from theoretical definitions of order and causal relationships.

What I would take back to an app project

In my RAG demo, making sources visible helps a reader inspect where an answer came from. It does not, by itself, establish that the answer describes events in a sensible sequence. Those are separate evaluation questions.

This paper does not give me a ready-made accuracy score or a proven hallucination detector. I have not tested its measurement in my own apps.

What it gives me is a sharper question: instead of only asking whether a generated explanation sounds good, what changes when I perturb its event order?

For me, that is the interesting bridge between theory and building. “Does this model represent the world?” is enormous. Choosing a controlled reversal and studying its probability makes one part of that question more concrete.

Sources

University of Tokyo press release (November 6, 2026): https://www.t.u-tokyo.ac.jp/press/pr2026-11-06-001
Takahiro Sagawa, Stochastic Thermodynamics for Autoregressive Generative Models: A Non-Markovian Perspective, PRX Intelligence. DOI: 10.1103/tv38-b23y: https://doi.org/10.1103/tv38-b23y

I Built a Japanese PDF RAG Demo Where You Can See What the Model Looked At

kohei matsunobu — Fri, 06 Nov 2026 17:57:20 +0000

This article was originally published in Japanese on Zenn.

Introduction

Hi, I'm Kohei Matsunobe, a second-year student in the online (correspondence) program at ZEN University. I'm working toward becoming an AI engineer, and I learn by building LLM-related systems with my own hands.

When you ask an LLM about a document, you usually get back an answer in fluent, plausible language. What bothered me was the question: "Which part of the document is that answer actually based on?" I didn't want to judge an answer by whether it sounded right. I wanted to build something where it's clear what the model looked at when it answered.

So I built a RAG demo: you upload PDFs, ask questions in Japanese, and get answers with sources (file name and page number).

https://github.com/koheip/rag-doc-qa

This post isn't a usage guide. Instead, it focuses on:

What I decided, and where, in order to show sources accurately
What I learned once I evaluated it and looked at the numbers

Setup instructions are in the README, so please check there.

What I built

It's a Streamlit app. You register PDFs and ask a question; it searches for relevant passages and answers using only those passages as evidence. Answers carry numbers like [1], and each number maps to "file name p.page". You can also see the retrieved chunks and their similarity scores in the UI.

Retrieval: multilingual-e5-small + FAISS (IndexFlatIP)
Generation: Google Gemini (official SDK google-genai). The default is gemini-3.5-flash-lite, and you can switch to gemini-3.8-flash from the sidebar
Without an API key, it runs in "extractive mode," which shows the retrieved passages as-is

Here's the flow:

flowchart LR
    A[PDF] -->|pypdf| B[Text per page]
    B -->|Japanese-aware chunking| C[Chunks]
    C -->|"passage: " + e5| D[(FAISS)]
    Q[Question] -->|"query: " + e5| D
    D --> R[top-k chunks + sources]
    R -->|with key| L[Answer with Gemini]
    R -->|no key| E[Extractive mode]

Decision 1: Chunks never cross page boundaries

The decision that mattered most for showing accurate page numbers was how I split text into chunks.

I wrote the chunker myself instead of using a library. Japanese has no spaces between words, so I measure length in characters. Then I split into roughly 200-character pieces, preferring sentence boundaries like 「。」, and overlap a bit of the previous chunk's tail so context isn't cut off at the seams.

And chunks never cross pages. If a chunk spans two pages, you can't say which single page it came from. Since the whole point of the app is showing sources, I prioritized citation accuracy over retrieval convenience here.

I wrote it myself because I wanted Japanese-specific splitting in a form I could actually explain. If you leave it to a library, it's harder to explain in your own words why the text was cut where it was.

Decision 2: Make it say "I don't know," and cite only what it used

In the generation prompt, I number the retrieved chunks, attach their sources, and give instructions like this (from src/llm.py; the prompt itself is in Japanese):

SYSTEM_PROMPT = """あなたは社内ドキュメントに関する質問に答えるアシスタントです。
以下のルールを必ず守ってください。
1. 回答は【参考資料】に書かれている内容だけを根拠にしてください。あなた自身の知識で補ってはいけません。
2. 【参考資料】に答えが書かれていない場合は、推測せずに「わからない」とだけ答えてください（この場合は資料番号を付けないでください）。
3. 回答の文末には、根拠にした資料の番号を [1] [2] のように付けてください。
4. 日本語で、簡潔かつ丁寧に答えてください。"""

Roughly translated, the rules are:

Base your answer only on the provided reference material; don't fill in with your own knowledge.
If the answer isn't in the material, don't guess — just say 「わからない」 ("I don't know"), with no reference numbers.
End the answer with the numbers of the passages you relied on, like [1] [2].
Answer in Japanese, concisely and politely.

The key point is that I don't list every retrieved chunk as a "source." Only the numbers the answer actually references are shown as sources. Even if the top 4 chunks are passed in, if the answer used just one, only one source is shown. When the answer is 「わからない」, no sources are shown.

Temperature is set to 0.1, because I wanted answers that stay faithful to the material and don't wobble between runs.

Decision 3: Let people try it without an API key

Without an API key, the app runs in "extractive mode," showing the top chunks directly as the answer.

I added this so anyone could try it right away, but once I built it I found another benefit. Extractive mode is retrieval only, so before generation even happens, you can check whether the right passages were found at all. When the RAG answer is off, it became much easier to tell whether the problem is in retrieval or in generation.

Other choices

Item	Choice	Why
UI	Streamlit	Python only; chat UI and file upload in a small amount of code
PDF loading	pypdf	Pure Python with few dependencies; behaves the same on Windows/Mac/Linux
Embeddings	multilingual-e5-small	Free, runs locally, supports Japanese. About 470MB and fast enough on CPU
Vector store	FAISS (IndexFlatIP)	Shows how to use a library that's standard in practice, and the index can be swapped as data grows
LLM	Gemini (google-genai)	Has a free tier and can be called in a few lines with the official SDK

E5 expects a passage: prefix on documents and a query: prefix on questions, so I follow that. Vectors are 384-dimensional and normalized to length 1, so FAISS's inner product is exactly cosine similarity.

At this data size, a numpy matrix multiplication would be plenty. But with FAISS, when data grows I can swap the brute-force IndexFlatIP for an approximate index. The vector store is isolated in one file, so I can change it without affecting the rest.

What the evaluation taught me

I didn't want to stop at "it kind of works," so I made retrieval and answers measurable. The results below are what I wrote up in the README on 2026-10-21.

Eval data: 30 questions on the bundled sample PDF (employment rules and an internal FAQ for a fictional company, 11 pages)
Of the 30, 26 are answerable from the document and 4 are not. It also includes paraphrased questions that use different words from the document
Retrieval is dense vector search (e5-small, top-k=4)

Retrieval: Hit@3 is high, Hit@1 is still low

With a chunk size of 200 characters (the one I adopted), retrieval on the 26 answerable questions looked like this:

Metric	Value
Hit@1	73.1%
Hit@3	96.2%
Recall@4	100.0%
MRR	0.856
Hit@3 on paraphrased questions	92.3%

Every correct passage is within the top 4, but it's ranked first only 73.1% of the time. In other words, the right answer often lands at rank 2–4.

Right now I pass the top 4 chunks to the LLM together, so this doesn't affect the answers. But if I want to pass fewer chunks to lower cost, Hit@1 needs to improve. The clearest thing the numbers showed me was that "the answer is correct" and "retrieval is good enough" are two different things.

Bigger chunks aren't automatically better

I compared chunk sizes of 150, 200, 300, and 400 characters, and 200 gave the best balance. The surprise was 300: Hit@3 on paraphrased questions dropped from 92.3% to 69.2%. My read is that one chunk ends up mixing several rules (articles), so the meaning of the vector gets blurry.

If I had chosen the size based only on the intuition that "more context should be better," I would have missed this.

Generation: this dataset doesn't separate models

With gemini-3.5-flash-lite (the default), all 26 answerable questions were answered correctly (26/26), and all 4 unanswerable questions were correctly refused with 「わからない」 (4/4). Latency was 0.56s at p50 and 0.72s at p95. Grading was done by a different LLM than the answering model, comparing each answer with a reference answer and scoring it 0/1 (LLM-as-a-judge).

I don't read this as "the model is perfect." I read it as the current dataset is too easy and has hit the ceiling. To see differences, I need harder questions — ones that span multiple passages, or that require calculation.

Limits of this evaluation

30 questions is small; a single question moves the score by about 4 points
Both the dataset and the grading come from one person (me) and one LLM
I also tried comparing against gemini-3.8-flash, but it was under heavy load (503) during evaluation and didn't respond, so I couldn't compare

Known limitations

Here are the limits of this demo that I found while building it:

E5 similarity scores tend to cluster around 0.8–0.9, so it's hard to set a threshold like "below X means unrelated." That's why, for questions not covered by the document, I leave the decision to the LLM via the prompt and have it say 「わからない」
With paraphrases like 給料 vs. 給与 (both meaning "salary"), the correct passage sometimes isn't ranked first
Scanned-image-only PDFs need OCR, so they're not supported
The index lives only in memory and disappears when the session ends (I prioritized simplicity for a demo)

Tests

Tests are written with pytest and cover chunking, retrieval, the LLM-related logic, and the evaluation metrics. Retrieval tests use a lightweight fake embedder built from character counts (FakeEmbedder), and generation tests use a fake LLM (FakeLLM), so you can check the main flow without downloading the model and without calling the API.

Next: hybrid search

To improve Hit@1, my next step is hybrid search that combines BM25 (keyword search) with vector search. Keyword search is better at proper nouns, numbers, and article numbers, so the idea is to pair it with vector search, which handles paraphrases better.

I haven't implemented it yet. I've made the retrieval method swappable and prepared the function specs and tests for src/bm25.py first. I plan to measure whether Hit@1 and MRR go up on the same 30 questions.

Wrap-up

To show accurate page numbers, chunks never cross pages, and I wrote my own Japanese-aware chunker
Only the numbers the answer actually references are shown as sources, and when the document doesn't cover the question, the model says 「わからない」
In evaluation, Hit@3 was 96.2% but Hit@1 was 73.1%. Widening chunks to 300 characters dropped paraphrase Hit@3 from 92.3% to 69.2%
"The answer is correct" and "retrieval is good enough" are different things, and I only saw what to do next once I put numbers on it

When building RAG, I found it helps to be able to check "what did it look at to answer?" both in the UI and in numbers, not just how the answer looks. All the code is public, so feel free to try it out.

https://github.com/koheip/rag-doc-qa

What I Thought About Before Putting a Generative-AI Shrine Guide in My Indie iOS App (API Gateway + Lambda + Gemini, Consent and Privacy)

kohei matsunobu — Fri, 06 Nov 2026 17:57:15 +0000

This article was originally published in Japanese on Qiita.

Introduction

Hi, I'm Kohei Matsunobe. I'm a second-year student in the online (correspondence) program at ZEN University, studying to become an AI engineer.

On my own, I build an iOS app called KamiMeg (カミメグ). It's a digital goshuin book: you take photos of the goshuin (the calligraphy stamps you receive at Japanese shrines), find shrines nearby, and keep a record of your visits. It's free, there's no account, and your records are stored on your device.

I released v1.0 in October 2026 and v1.1.0 in November 2026. The headline feature of v1.1.0 is a generative-AI feature where a "companion deity" introduces the shrine to you when you press a button.

This post is about what I thought through before shipping that feature. It's not a code walkthrough; it's about three decisions:

Where the API key should live (and why I went serverless)
How to keep the app's promise — "your records stay on your device" — even after adding AI
Why I ended up with Gemini, even though I first looked at Bedrock and the Claude API

KamiMeg before the AI feature

Originally, KamiMeg was an app that did almost everything on the device.

Role	What I use
App	Expo / React Native (TypeScript)
Storing records	SQLite (on device)
Finding nearby shrines	Google Places API
Ads	Google AdMob

There was no backend server of my own and no account. "Your goshuin photos and visit records live only on your own device" was something I really cared about as the person building it.

When I submitted v1.0, App Store review needed some extra back-and-forth. I responded, it passed, and the app went live — but it taught me that even as an indie developer, once you ship to a store, you should build things you can explain. That became a starting assumption when I designed the AI feature.

What I wanted to build

When you're looking for shrines and logging visits, there are moments where you wonder, "What kind of place is this shrine?" So I thought it would be fun to show a short introduction for each shrine. Just showing a plain description felt a bit flat, though, so in the app it's the "companion deity" who introduces the shrine.

The introduction is never generated automatically. It's generated only when the user presses the button. I'll explain why in the privacy section.

In v1.1.0 I also made it so that when you search for a shrine by name, only that shrine is shown, and I unified the app's display name as "カミメグ".

Decision 1: Don't put the API key in the app (API Gateway → Lambda → Gemini)

The simplest way to use a generative-AI API from an app is to call it directly from the app. But that means shipping the API key inside the app. A key inside an app binary can be extracted, and since I manage the spending cap myself as an indie developer, I wanted to avoid that.

So I put a thin AWS serverless layer between the app and the AI.

flowchart LR
    A[iOS app<br/>KamiMeg] -->|Shrine name & address<br/>after consent only| B[Amazon API Gateway]
    B --> C[AWS Lambda]
    C --> D[Google Gemini API]
    D --> C
    C --> B
    B -->|Introduction text| A

The flow is simple: app → Amazon API Gateway → AWS Lambda → Google Gemini API. The Gemini API key lives only on the server side; the app never has it.

Why serverless:

I didn't want an always-on server. Introductions are generated only when the button is pressed, so it's enough for something to run only when a request comes in.
I can swap what's behind it without changing the app. The app only talks to API Gateway, so which AI the Lambda calls can change without an app update. This turned out to matter a lot when I switched to Gemini (more on that below).
I can enforce usage limits on the server. A daily usage limit can live on the server side (see below).

Decision 2: Don't break the "records stay on your device" promise

Adding an AI feature means sending something off the device. KamiMeg's selling point is "no account, records on device," so this is where I was most careful.

Here are the rules I settled on:

Only the shrine name and address are sent. That's the minimum needed to write an introduction.
They're sent only after the user consents. Without consent, the AI feature isn't used.
Goshuin photos, visit records, and location are not sent.
No account. I don't ask users to log in for the AI feature either.
There's a daily usage limit. It's tracked with a random per-device ID, which isn't information that identifies a person.
Generated introductions may be stored on the developer's server for up to 90 days. This is stated explicitly in the privacy policy (PRIVACY.md).

"Generate only when the button is pressed" is part of these rules too. I didn't want a behavior where just opening a screen silently sends data somewhere. Data that the user agreed to share is sent only when they press the button themselves. I think that's easier to explain, and it's more reassuring for the people using the app.

The usage limit also protects my costs. In indie development, if an API keeps getting called unexpectedly, that bill is on me. Being able to hold the limit on the server is another reason I'm glad the app doesn't call the API directly.

Decision 3: Why Gemini

To be honest, I didn't decide on Gemini from the start. At first I also considered Amazon Bedrock and the Claude API as candidates.

But once I actually moved forward, I found that usage limits and payment conditions didn't fit my situation as an indie developer, and I couldn't take it to production as-is. So I switched to the Gemini API, finished the feature, and shipped it to production.

This is where Decision 1 saved me. Because the app only knows about API Gateway, changing which AI gets called was a matter inside the Lambda. Leaving room to change "which AI to use" later is, I think, the reason the switch didn't stall the release.

The lesson I took from this: before comparing model quality, check early whether you can actually use the service in production from your own position. Limits and payment conditions are hard to see when you just call an API a few times to try it out.

Decide what you won't publish

When releasing the app and writing about it, I also decided what not to publish: the API endpoint URL, the app's key, the AWS account and region, and specific numbers for usage limits. None of them appear in this post.

Showing the flow "API Gateway → Lambda → Gemini" in a diagram is fine for explaining how it works. Concrete endpoints and limit values, on the other hand, could become clues for abuse. As an indie developer you sometimes feel like making everything open, but this is where I draw the line.

Caveats and what's next

AI-generated introductions aren't guaranteed to be accurate. As the builder, I see them as an "entry point," and I think it's best to check the details with the shrine's official information.
Because introductions are generated through the server, the feature assumes the user has consented and is online.
Going forward, I want to keep adding features without breaking the app's promises: no account, records stay on the device.

Wrap-up

I didn't call the generative-AI API directly from the app. A serverless setup — API Gateway → Lambda → Gemini — keeps the key and the usage limit on the server.
Only the shrine name and address are sent, and only after consent. Photos, records, and location are not sent, and there's no account.
I considered Bedrock and the Claude API, but switched to Gemini because of limits and payment conditions. Since the app doesn't know what's behind the API, the switch didn't stop the release.

When you put generative AI into an app, I found it pays off to decide "what you send and what you don't" and "whether you can swap it later" before picking a model. I hope this is useful for anyone else building apps on their own.

KamiMeg is available on the App Store (Japanese). If you like visiting shrines, give it a try:
https://apps.apple.com/jp/app/カミめぐ/id6821107881

Before You Burn GPU Hours, Smoke-Test One Task on a Cheap CPU Runtime

kohei matsunobu — Sat, 31 Oct 2026 07:34:56 +0000

This article was originally published in Japanese on Zenn.

Introduction

Hi, I'm Kohei Matsunobe, a second-year student in the online (correspondence) program at ZEN University. I'm working toward becoming an AI engineer, and on my own I run experiment pipelines that evaluate LLM outputs and code on notebooks and cloud runtimes.

GPU runtimes are great, but they're expensive per hour, and there's a cap on how many compute units you can use. I've made the same mistakes more than once: "I grabbed a GPU, launched the real job, and it crashed in the first few minutes," or "it crashed, I didn't notice, and the runtime just sat there running."

Looking back, almost all of those failures could have been caught without a GPU. In this post, I'll walk through the idea of a "smoke test" — a minimal check you run on a cheap CPU runtime before moving up to the expensive one — plus a small script you can use right away.

What tends to go wrong

In pipelines that evaluate LLMs or generated code, these problems come up a lot:

Missing test dependencies: Your setup grades generated code with pytest, but the production runtime doesn't have pytest, or the libraries the tested code needs.
Cache or data not found: The path where you thought you'd put model weights or dataset caches is empty on the fresh runtime. So downloads run every time, or you get errors because offline mode is still on.
Mismatched environment variables or paths: It worked locally, but on a different runtime the working directory or cache location has changed.
An exception on the very first task: Simple mistakes, like a missing output directory or a misspelled argument name.

None of these have anything to do with GPU performance. But when they happen on a GPU runtime, you lose all the time spent on setup and model loading too. Retry the job enough times and you might suddenly find you've used up your compute units.

The idea behind the smoke test

It boils down to three points:

Keep dependencies identical to production: Build the environment from the same definition as production — the same requirements file, lock file, or container image. If you maintain a separate dependency list for CPU, you hide the very differences you want to catch. (The only exception: packages that need a GPU-specific build get handled separately.)
Run just one task on the cheap runtime: Don't run the whole eval set — just one item. Inference may be slow on CPU, but all you need to know is "does it make it to the end?"
Promote to the expensive runtime only if it passes: Only when the smoke test succeeds do you run the real job on the GPU runtime, with the same code and the same config.

If you think of it as a CI pipeline, it's a staged gate: put a light job in front of the heavy one, and don't move on unless the light one passes.

The minimum checklist

Here's what I check every time:

Can the key packages be imported? (Including pytest for evaluation)
Do the cache and data directories exist, and are they non-empty?
Are the required environment variables set?
Can I write to the output directory?
Does at least one pytest test actually pass?
Can the same entry point as production run to completion as a dry run with just one task?

You don't need to do all of these perfectly. Just adding "anything that has bitten me even once" to the list goes a long way.

A small script

Here's a script that bundles the checks above. Adjust REQUIRED_MODULES, REQUIRED_PATHS, and run_task to fit your own pipeline.

#!/usr/bin/env python3
# smoke_test.py
"""Smoke test to run on a cheap runtime before moving to an expensive one."""
from __future__ import annotations

import argparse
import importlib
import json
import os
import subprocess
import sys
from pathlib import Path

# Modules to verify with the same dependency stack as production
REQUIRED_MODULES = ["numpy", "pytest", "transformers"]
# Cache/data paths that should exist and have contents
REQUIRED_PATHS = [
    Path(os.environ.get("HF_HOME", "~/.cache/huggingface")).expanduser(),
    Path("data"),
]
REQUIRED_ENV = ["HF_HOME"]


def check_imports(modules: list[str]) -> list[str]:
    errors = []
    for name in modules:
        try:
            importlib.import_module(name)
        except Exception as e:  # also catch init errors, not just ImportError
            errors.append(f"import failed: {name} ({type(e).__name__}: {e})")
    return errors


def check_paths(paths: list[Path]) -> list[str]:
    errors = []
    for p in paths:
        if not p.exists():
            errors.append(f"path missing: {p}")
        elif p.is_dir() and not any(p.iterdir()):
            errors.append(f"directory is empty: {p}")
    return errors


def check_env(names: list[str]) -> list[str]:
    return [f"env var not set: {n}" for n in names if not os.environ.get(n)]


def check_writable(out_dir: Path) -> list[str]:
    try:
        out_dir.mkdir(parents=True, exist_ok=True)
        probe = out_dir / ".smoke_write_test"
        probe.write_text("ok", encoding="utf-8")
        probe.unlink()
    except OSError as e:
        return [f"not writable: {out_dir} ({e})"]
    return []


def run_one_pytest(test_id: str) -> list[str]:
    # e.g. tests/test_eval.py::test_single_case
    result = subprocess.run([sys.executable, "-m", "pytest", "-q", "-x", test_id])
    # Exit code 5 means "no tests were collected", so treat it as a failure too
    if result.returncode != 0:
        return [f"pytest failed: {test_id} (exit={result.returncode})"]
    return []


def load_tasks(path: Path) -> list[dict]:
    with path.open(encoding="utf-8") as f:
        return [json.loads(line) for line in f if line.strip()]


def run_task(task: dict) -> dict:
    # Replace this with the same processing as production (inference, grading)
    return {"id": task.get("id"), "status": "ok"}


def main() -> int:
    parser = argparse.ArgumentParser(description="cheap runtime smoke test")
    parser.add_argument("--tasks", type=Path, help="JSONL file, one task per line")
    parser.add_argument("--out", type=Path, default=Path("outputs"))
    parser.add_argument("--pytest", metavar="TEST_ID", help="run a single test ID")
    parser.add_argument("--dry-run", action="store_true", help="run only one task")
    args = parser.parse_args()

    errors: list[str] = []
    errors += check_imports(REQUIRED_MODULES)
    errors += check_paths(REQUIRED_PATHS)
    errors += check_env(REQUIRED_ENV)
    errors += check_writable(args.out)
    if args.pytest:
        errors += run_one_pytest(args.pytest)

    if errors:
        print("[NG] Pre-flight checks found problems:")
        for e in errors:
            print("  -", e)
        return 1

    if args.tasks:
        tasks = load_tasks(args.tasks)
        if args.dry_run:
            tasks = tasks[:1]
        for task in tasks:
            result = run_task(task)
            print(json.dumps(result, ensure_ascii=False))

    print("[OK] Smoke test passed. Safe to promote to the expensive runtime.")
    return 0


if __name__ == "__main__":
    sys.exit(main())

On the cheap CPU runtime, run it like this:

python smoke_test.py --tasks data/tasks.jsonl --dry-run \
  --pytest tests/test_eval.py::test_single_case

If you only start the real run when the exit code is 0, the same pattern works in notebooks and in CI:

python smoke_test.py --tasks data/tasks.jsonl --dry-run && python run_full_eval.py

Wrap-up

Most expensive GPU job failures come from things that have nothing to do with the GPU: dependencies, caches, and path settings.
Keep the same dependency stack as production, push just one task through on a cheap CPU runtime, and promote to the expensive runtime only when it passes.
For the checklist, adding things you've personally been bitten by, little by little, is plenty.

A few minutes of smoke testing protects both your compute units and your motivation. I hope this helps anyone else running experiments on their own!

Summarize Kaggle Notebook Status and Logs in ~7 Lines with the Kaggle CLI (Goodbye, .bat + Notepad)

kohei matsunobu — Sat, 31 Oct 2026 07:34:55 +0000

This article was originally published in Japanese on Qiita.

Introduction

Hi, I'm Kohei Matsunobe. I'm a second-year student in the online (correspondence) program at ZEN University, studying to become an AI engineer.

Once I started running Kaggle notebooks (kernels) from my local machine with kaggle kernels push, one small thing kept eating my time: checking what was going on. Is it still running? Did it finish? Did it fail? And if it failed, where? Checking that in the browser or digging through log files every single time got old fast.

In this post, I'll share a script that calls kaggle kernels status and kaggle kernels output from Python and boils the status and log down to a ~7-line summary plus one CSV row. I'll also cover how I fixed garbled text (mojibake) on Windows, and how I made the same script usable both from a .bat file and from an AI agent.

Note: this post is not about submissions. It focuses purely on monitoring run status and summarizing logs.

How I used to do it

At first, I got by with a batch file like this:

@echo off
kaggle kernels status koheip/my-notebook
kaggle kernels output koheip/my-notebook -p .\out -o
notepad .\out\my-notebook.log

It worked, but it had a few problems:

The logs are long, and scrolling through Notepad to find the error is painful
Non-ASCII output (Japanese, in my case) got garbled in the console
There was no record of "when, which notebook, and what status"
When I asked an AI agent to check the log, it had to read the entire thing, which is wasteful

That last point was the big one. If you're handing something to an agent, a short text with only the information you need is much easier to work with.

What the CLI gives you

You only need two commands (official docs: Kaggle CLI documentation, kaggle.com/docs/api).

`kaggle kernels status`

Shows the status of the latest run. You pass the kernel as owner/kernel-slug.

kaggle kernels status koheip/my-notebook

The output is a single line (on failure, an extra Failure message: "..." line is added):

koheip/my-notebook has status "KernelWorkerStatus.COMPLETE"

Depending on the CLI version, you may see lowercase complete or the KernelWorkerStatus. prefix, so the script takes only the last word and lowercases it.

`kaggle kernels output`

Downloads the output files from the latest run. The main options:

Option	Meaning
`-p, --path`	Destination folder
`-o, --force`	Overwrite existing files
`-q, --quiet`	Suppress progress output
`--file-pattern`	Only download files matching a regex

The key point: along with the output files, it also saves the execution log as <kernel-slug>.log. In most cases, it's a JSON array of entries with stream_name (stdout / stderr), time, and data. That's what we'll read and summarize.

If your notebook produces large output files, narrowing things down with --file-pattern speeds it up.

Authentication

The Kaggle CLI can authenticate via the KAGGLE_API_TOKEN environment variable, ~/.kaggle/access_token, or the classic ~/.kaggle/kaggle.json, among others. I went with reading the token from a dedicated file and passing it as an environment variable only to the child process. That way, whether the script is called from a .bat file or from an agent, the token is loaded the same way.

The summary script

Save this as kaggle_status_summary.py. It only uses the standard library.

# kaggle_status_summary.py
import argparse
import csv
import json
import os
import re
import subprocess
import sys
from datetime import datetime
from pathlib import Path

# Force stdout to UTF-8, even on Windows (Python 3.7+)
if hasattr(sys.stdout, "reconfigure"):
    sys.stdout.reconfigure(encoding="utf-8", errors="replace")

DEFAULT_TOKEN_FILE = Path.home() / ".config" / "kaggle_token.txt"


def build_env(token_file: Path) -> dict:
    """Read the token from a file and build env vars for the child process."""
    env = os.environ.copy()
    env["PYTHONIOENCODING"] = "utf-8"
    env["PYTHONUTF8"] = "1"
    if token_file.exists():
        env["KAGGLE_API_TOKEN"] = token_file.read_text(encoding="utf-8").strip()
    return env


def run(cmd: list[str], env: dict) -> str:
    r = subprocess.run(
        cmd, env=env, capture_output=True,
        text=True, encoding="utf-8", errors="replace",
    )
    if r.returncode != 0:
        raise RuntimeError(f"{' '.join(cmd)} failed: {r.stderr.strip()[:300]}")
    return r.stdout


def get_status(kernel: str, env: dict) -> tuple[str, str]:
    out = run(["kaggle", "kernels", "status", kernel], env)
    m = re.search(r'has status "([^"]+)"', out)
    status = m.group(1).split(".")[-1].lower() if m else "unknown"
    f = re.search(r'Failure message: "(.*)"', out)
    return status, (f.group(1) if f else "")


def read_log(log_path: Path) -> list[tuple[str, str]]:
    """Return a list of (stream_name, text). Fall back to plain text if it isn't JSON."""
    raw = log_path.read_text(encoding="utf-8", errors="replace")
    try:
        items = json.loads(raw)
        return [(i.get("stream_name", ""), str(i.get("data", ""))) for i in items]
    except (json.JSONDecodeError, AttributeError, TypeError):
        return [("", line) for line in raw.splitlines()]


def main() -> int:
    ap = argparse.ArgumentParser()
    ap.add_argument("kernel", help="owner/kernel-slug")
    ap.add_argument("--out", default="kaggle_out")
    ap.add_argument("--csv", default="kaggle_status_log.csv")
    ap.add_argument("--token-file", type=Path, default=DEFAULT_TOKEN_FILE)
    args = ap.parse_args()

    env = build_env(args.token_file)
    status, failure = get_status(args.kernel, env)

    slug = args.kernel.split("/")[1]
    out_dir = Path(args.out) / slug
    entries, files = [], []
    if status in ("complete", "error", "cancelacknowledged"):
        out_dir.mkdir(parents=True, exist_ok=True)
        run(["kaggle", "kernels", "output", args.kernel,
             "-p", str(out_dir), "-o", "-q"], env)
        files = [p.name for p in out_dir.iterdir() if p.is_file()]
        log_path = out_dir / f"{slug}.log"
        if log_path.exists():
            entries = read_log(log_path)

    err_pat = re.compile(r"(Error|Exception|Traceback)")
    stderr_n = sum(1 for s, _ in entries if s == "stderr")
    err_lines = [t.strip() for _, t in entries if err_pat.search(t)]
    last_line = next((t.strip() for _, t in reversed(entries) if t.strip()), "")
    now = datetime.now().strftime("%Y-%m-%d %H:%M:%S")

    # The "~7 line" summary
    print(f"[kernel]  {args.kernel}")
    print(f"[checked] {now}")
    print(f"[status]  {status}" + (f" / {failure}" if failure else ""))
    print(f"[log]     {len(entries)} entries (stderr: {stderr_n})")
    print(f"[errors]  {len(err_lines)} lines")
    print(f"[last_err] {err_lines[-1][:120] if err_lines else '-'}")
    print(f"[tail]    {last_line[:120] or '-'}  / files: {len(files)}")

    # Append one row to a CSV history (with BOM so Excel opens it cleanly)
    csv_path = Path(args.csv)
    new = not csv_path.exists()
    with csv_path.open("a", newline="", encoding="utf-8-sig") as f:
        w = csv.writer(f)
        if new:
            w.writerow(["checked_at", "kernel", "status", "log_entries",
                        "stderr", "error_lines", "last_error", "files"])
        w.writerow([now, args.kernel, status, len(entries), stderr_n,
                    len(err_lines), err_lines[-1][:200] if err_lines else "",
                    len(files)])
    return 0 if status != "error" else 1


if __name__ == "__main__":
    sys.exit(main())

Running it looks like this (values are just an example):

[kernel]  koheip/my-notebook
[checked] 2026-10-31 15:42:10
[status]  complete
[log]     214 entries (stderr: 12)
[errors]  0 lines
[last_err] -
[tail]    Done.  / files: 3

Three things I was deliberate about:

Exit code: it returns 1 when the status is error, so it's easy to branch on with if errorlevel 1 in a .bat file or on the agent side
Status normalization: to absorb differences between CLI versions, KernelWorkerStatus.COMPLETE becomes complete
No download while running: for running / queued, it only checks the status and exits, so it stays lightweight even if you call it often

That said, the Kaggle API has rate limits. If you monitor in a loop, leave a gap of a few minutes between checks.

Setting up the token file

Generate a token on your Kaggle settings page and save it as a one-line text file.

# PowerShell: create the token file (it should contain only the token, on one line)
New-Item -ItemType Directory -Force "$HOME\.config" | Out-Null
Set-Content -Path "$HOME\.config\kaggle_token.txt" -Value "YOUR_TOKEN_HERE" -Encoding ascii -NoNewline

I'm strict about two rules: never put this file under Git, and never paste it into an article or a chat. When an agent uses the script, all it needs is the script's path; it never has to see the token itself.

Fixing garbled text on Windows

On Windows, the console code page often isn't UTF-8 by default (on Japanese Windows it's CP932 / Shift_JIS). Kaggle logs, on the other hand, are UTF-8. If you do nothing, non-ASCII characters get garbled, or the script dies with a UnicodeEncodeError.

The fix is to switch to UTF-8 on both the console side and the Python side.

Calling from a .bat file

@echo off
chcp 65001 >nul
set PYTHONIOENCODING=utf-8
set PYTHONUTF8=1
python "%~dp0kaggle_status_summary.py" koheip/my-notebook
if errorlevel 1 echo [!] The run ended with an error

If your .bat file itself contains non-ASCII text, save it as UTF-8 so it displays correctly after chcp 65001.

Calling from PowerShell

chcp 65001 > $null
[Console]::OutputEncoding = [System.Text.Encoding]::UTF8
$OutputEncoding = [System.Text.Encoding]::UTF8
$env:PYTHONIOENCODING = "utf-8"
$env:PYTHONUTF8 = "1"

python .\kaggle_status_summary.py koheip/my-notebook
if ($LASTEXITCODE -ne 0) { Write-Host "[!] The run ended with an error" }

What the Python side does

Inside the script, I also do three things just to be safe:

sys.stdout.reconfigure(encoding="utf-8") to make stdout UTF-8
subprocess.run(..., encoding="utf-8", errors="replace") to read the child process output as UTF-8
Pass PYTHONIOENCODING=utf-8 to the child process (the Kaggle CLI) as well

There are cases where chcp 65001 alone, or PYTHONIOENCODING alone, only fixes half of the problem. That tripped me up at first. Set both and you're good.

Wrap-up

kaggle kernels status gives you the status; kaggle kernels output gives you the output files and <slug>.log
Reading the log in Python and condensing it into a ~7-line summary plus one CSV row makes checking much easier
Loading the token from a file and passing it to the child process means .bat files and agents use the same path
On Windows, use both chcp 65001 and PYTHONIOENCODING=utf-8 to avoid garbled text

Compared to the days of scrolling through logs in Notepad, I now spend far less time checking runs, and I hand far less text to my agents. I hope this helps anyone else running Kaggle notebooks from their local machine!

Eliminating Accidental Submissions in Kaggle Work: The Kaggle CLI, a Confirmation Wrapper, and Measuring on Production Hardware

kohei matsunobu — Sat, 17 Oct 2026 10:56:10 +0000

This article was originally published in Japanese on Zenn.

Introduction

I've been working on Kaggle with help from LLM agents and scripts. Automation makes things easier, but it also nearly caused accidents, like the workflow going all the way to submission without anyone checking.

In this article, I'll cover how to work with notebooks using the Kaggle CLI, a wrapper that always puts a human confirmation step in the loop, and what I learned from the differences between my validation environment and the production environment.

1. Checking notebook status with the Kaggle CLI

Setting up the token

With the newer token method, you put your token in the KAGGLE_API_TOKEN environment variable.

export KAGGLE_API_TOKEN="(your token issued by Kaggle)"

Getting notebook status and output

You can check the status of a notebook running on Kaggle and fetch its output from the CLI.

# Check whether it's running, finished, or failed
kaggle kernels status <username>/<notebook-name>

# Download the output files locally
kaggle kernels output <username>/<notebook-name>

Since you can see the status without opening a browser, it's much easier to handle from scripts and agents.

2. A confirmation wrapper to prevent accidental submissions

I once had an automated batch go all the way to submission without any confirmation. Submissions are often limited in number, and they can't be undone.

So I built a wrapper that splits the path to submission into these steps:

Show which file will be submitted
Confirm with y/N (pressing just Enter means "No")
Run it
Validate the output (whether the file exists, whether the format is correct, etc.)
Only display the submit command and stop there

For the final submission, a human runs the displayed command themselves. This way, even if something breaks somewhere in the automation, a human always takes a look right before submission.

The confirmation part can be written in Python in just a few lines, for example:

def confirm(message: str) -> bool:
    answer = input(f"{message} [y/N]: ").strip().lower()
    return answer == "y"

if not confirm("Run with this file?"):
    raise SystemExit("Aborted")

The key is making "No" the default. Even when it's called from an agent or script with no input, it won't proceed on its own.

3. Measure time-limited workloads on the same hardware as production

When I was doing local validation on a fast GPU, the trends in the results sometimes differed from those on the slower GPU used in production.

When a workload has a time limit, slower hardware alone can cause it to be cut off partway, or change what you can get done within the limit. Just because something "fits in time" on a fast environment doesn't mean it will in production.

I learned that for time-limited workloads, it's important to measure on the same hardware as production.

Summary

Once you set KAGGLE_API_TOKEN, the Kaggle CLI lets you track your notebooks with kernels status and kernels output, no browser needed
Even with automation, always put a human confirmation step right before submission (default to "No", and only display the submit command)
Measure time-limited workloads on the same hardware as production

The more I automate, the more I feel it's important to decide up front "where a human steps in to stop things."

I Stopped Feeding Whole Jupyter Notebooks to LLMs and Cut Tokens by Over 95% (Claude Code / Codex CLI / PowerShell)

kohei matsunobu — Sat, 17 Oct 2026 10:56:09 +0000

This article was originally published in Japanese on Qiita.

Introduction

When I work on Kaggle, I write code in Jupyter notebooks and ask Claude Code or Codex CLI for reviews and advice along the way. The problem: passing a whole notebook to an LLM came to about 30k tokens per notebook, and after asking a few times, the cost added up fast.

In this article, I'll share the tricks I used to save tokens, plus the gotchas I hit when calling these CLIs from Windows PowerShell.

1. Pass only the "code cell diff" of the notebook

An .ipynb file is JSON under the hood, and it's packed with cell outputs (logs and images) and metadata. In most cases, the code is all I actually want the LLM to look at, so I wrote a script that does the following:

Drops outputs and metadata, and extracts only the code cells
Outputs only the diff against the previous version
Replaces long lines over 300 characters with their "length" and a "hash" (all I need to know is whether the content changed)

With this, comparing versions fits in about 1–1.6k tokens (a reduction of over 95%).

Also, long prompt strings inside a notebook make it hard to read, so I create a "reading view" where those strings are split out into separate .txt files. The notebook I actually run stays as a single file, and I pass the reading view to the LLM or for reviews. That's how I keep the two roles separate.

2. Calling Claude Code / Codex CLI as commands from another program

Claude Code

With claude -p, you can ask a single question and get an answer back without opening the interactive UI.

claude -p --max-turns 1 --disallowedTools "Bash,Read,..."

The key is to disallow tools like file reading and pass only the summary you've prepared yourself. Since it won't go off and read files in your repository on its own, token usage stays cheap.

Codex CLI

I installed it with npm.

npm i -g @openai/codex

When logging in through the browser failed, authenticating with a URL and a code worked.

codex login --device-auth

I pass the question via standard input.

echo "Your question here" | codex exec --skip-git-repo-check -

3. Japanese text gets garbled in Windows PowerShell

When I piped a Japanese prompt from PowerShell into the CLI, the text came out garbled (mojibake). Setting the following three to UTF-8 fixes it.

[Console]::InputEncoding  = [System.Text.Encoding]::UTF8
[Console]::OutputEncoding = [System.Text.Encoding]::UTF8
$OutputEncoding           = [System.Text.Encoding]::UTF8

Also, PowerShell doesn't let you chain commands with & the way cmd does; you get an AmpersandNotAllowed error. If you write commands with a cmd mindset, this will trip you up, so watch out.

Summary

Instead of passing the whole notebook, passing only the code cell diff cut tokens by over 95%
With claude -p, disallowing tools and passing only a summary keeps it cheap
If Codex CLI login fails, try codex login --device-auth
When passing Japanese text in PowerShell, set the three encodings to UTF-8

I hope this helps anyone else who's going back and forth between LLMs and notebooks like I am.