I Use AI to Write Most of My Code. So I Built a Tool That Tests Whether I Understand It.

#programming #ai #python #showdev

I'll be honest about something a lot of developers won't say out loud: AI writes most of my code.

I'm in a career change phase - eight months ago I was running operations for a community basketball association. Today I have six deployed ML and analytics projects. That speed was only possible because I leaned hard on LLMs. I describe what I want, the model writes it, I review, I iterate, I ship.

But somewhere in that loop, I noticed something that bothered me. Not a bug. Not a failure. A pattern: I'd tell Claude to do something, watch the green ticks come up, and mentally file it as work I had done.

But had I? The tests passed. The app ran. However, if you'd asked me to walk you through how it really worked, or even just a skeleton outline of how code blocks interacted with each other, I'd have been reading my own code like a stranger.

That's where I drew a line for myself: I can't say I've built something if I can't describe how it works. Shipping it isn't the same as owning it. There's a real ethical dimension to it - putting code into the world that you don't understand, and taking credit for it, is a quiet kind of dishonesty. To yourself, but also to people who will use what you are shipping.

It's also fairly dangerous - if you don't know how something works, and how it should work, how can you know if what the LLM produced is efficient and staying within your legal/ethical boundaries?

There's a practical dimension too. AI is already capable beyond what I can do alone - that gap is only growing bigger every day, especially with Fable 5 dropping just this week. This means that the skill that actually matters, the one I need to break into this industry, isn't producing code. It's understanding what's been produced, and knowing whether it's doing it correctly. Capability is cheap. Comprehension is the differentiator now.

So I built a tool to hold myself to that standard.

OwnYourShip (`oys`)

OwnYourShip is a CLI tool that scans your codebase and quizzes you on your own code.

The workflow is simple:

Run oys in your project directory
It scans your codebase and builds a picture of what's in it
It generates quiz questions about your functions, your logic, your design decisions
You answer. It tracks what you actually understand — and what you've been pretending to.

It's a Python CLI talking to a local FastAPI server, with SQLite tracking your progress over time. Question generation runs on Claude (Anthropic's API, your own key - nothing leaves your machine except the code snippets being quizzed). Quiz modes run from Easy through to a Tailored adaptive mode that focuses on the areas where your answers have been weakest.

The name is the thesis: you shouldn't ship what you can't explain.

What building it taught me

The irony isn't lost on me that I used AI heavily to build a tool about AI-assisted coding accountability. But that's exactly the point - the tool exists because I work this way, not in spite of it. I don't want to use AI less. I want my drive to produce projects to also drive my growth, instead of quietly replacing it.

A few things I took away:

The gap is real and it's silent. When the quiz hit functions I'd "written" 3 sessions earlier, my hit rate was worse than I expected. Green ticks don't measure understanding - it was frankly a bit embarrassing.

Explaining is a different skill from reviewing. Reading AI output and nodding along feels like understanding. Generating an explanation from scratch (text mode), or sorting through 4 feasible multiple choice options (MC mode) is a much more effective method.

The fix is cheap. Ten minutes of getting quizzed on a repo before an interview does more for my confidence than an hour of re-reading the code.

Honest limitations

It's early. The Python scanner is the most accurate (AST-based); other languages get lighter treatment for now. Question quality varies with codebase complexity. There's a roadmap in the README.

But it works, it's open source (MIT), and if you build with AI and you've ever felt that quiet gap between what you shipped and what you understand — it was built for you.

GitHub: https://github.com/JemHRice/ownyourship

I'd genuinely love feedback, issues, or brutal questions about my own code. That's sort of the whole idea.