What I Actually Learn When I Build With AI

I've spent entire nights building something with AI, gotten a bunch of features working, and then realized I couldn't really explain half of what I'd just built.

And then there are nights when I spend hours trying to figure out one stupid little problem, and somehow end up learning more from that than I did from the entire night before.

It's kind of funny when you think about it. Both times I was productive. Both times I ended up with working code. But what I actually learned from each experience was completely different.

I wrote a post recently about how AI is making it way too easy to avoid thinking, and it got me thinking about my own habits. Because I use AI coding tools constantly. I love them. They've helped me build things I probably wouldn't have even attempted otherwise.

But I'm also still learning. A lot.

And there is a big difference between just getting something working and actually understanding how it works.

Just because I understand the explanation doesn't mean I understand the code

This probably happens to all of us more than we'd like to admit.

I might ask AI to explain something, and it'll give me a perfectly reasonable breakdown. Everything makes sense. I can follow along, I understand what each part is doing, and I think, okay, got it.

Then 3 days later I need to change something, and suddenly I'm staring at the code like I've never seen it before.

Apparently, I understood it just fine when someone was holding my hand through the entire explanation.

That is not quite the same thing as actually knowing it, though.

And I'm not talking about memorizing syntax or remembering every little API detail. I look things up all the time, and I don't see anything wrong with that.

I'm talking about understanding why something was done a certain way.

If all I can remember about a feature is that AI wrote it and it worked, I haven't really learned much from it.

Sometimes the interesting part is the problem you didn't know existed

I've been working on a text editor called Slate, which has been a really interesting learning experience for me.

One thing I hadn't given much thought to before was what actually happens when you save a file while you're still editing it.

I mean, saving a file sounds pretty simple, right? Write the contents to disk, mark it as saved, done.

Except what happens if the user keeps typing while that save is happening?

The file being written contains the older content, but the editor now has newer changes. So if the save finishes and the editor clears the unsaved indicator, it's basically telling the user everything is saved when it isn't.

Slate uses revision-based save tracking to handle this. When a save starts, it remembers which revision of the document it's saving. That way, any changes made while the save is happening don't accidentally get marked as saved too.

Now, AI could have just implemented all of that for me. I could have tested the save button, seen that it worked, and moved on with my life.

But understanding why that revision tracking matters is what makes it useful to me.

It's not really about the code itself. It's about recognizing that something can change between the time an operation starts and the time it finishes.

And that's a problem that could come up in all kinds of applications, not just text editors.

That's the kind of thing I want to take away from a project.

Actually experimenting makes a huge difference

I've noticed the same thing when working with local AI models.

Let's say I run a model and it takes 30 seconds to complete something.

Okay, so it's slow. Right?

Well, maybe. But what actually took 30 seconds?

Was most of that time spent loading the model into memory? Was the actual processing slow? Would running it a second time be significantly faster?

Those are all things I can test.

And I find that I learn a lot more when I actually start experimenting instead of just asking AI to explain what's happening.

Run it again. Measure things separately. Change something. See what happens.

Sometimes the results are exactly what I expected. Other times they're not even close, which is usually when things get interesting.

I love that I can use AI to help me figure things out. I can ask what else I should try, or why the result isn't what I expected.

That makes learning unfamiliar things so much easier.

But there's a big difference between AI telling me why something should happen and me actually seeing what does happen.

And sometimes AI's explanation is wrong, too. Which I am sure we all know.

I don't necessarily remember the code, but I remember the problem

I think this is probably the biggest thing I've noticed.

When I really spend time understanding something, I might not remember the exact implementation later. But I remember the problem I was trying to solve and why I solved it that way.

And that carries over into other projects.

Maybe I start wondering whether some asynchronous operation could finish after the user has already changed something.

Or whether a performance issue is actually a startup issue.

Or whether I'm storing some piece of state that I could just calculate from something else.

I might still need to look things up. I might still ask AI to help me write the implementation.

But at least I know what I'm looking for and why it matters.

If I just accept the generated code without thinking much about it, I miss all of that. The solution is sitting right there in my project, but I might not even recognize the same problem when I run into it somewhere else.

And considering how often I start new projects, there's a pretty good chance I will run into it somewhere else. Probably next week.

Of course, I don't need to turn everything into a learning experience

I also think there's a balance here.

Sometimes I just want AI to fix the padding on a button.

I don't need to spend 45 minutes investigating the underlying principles of CSS spacing. I just want the button to look right.

And there are plenty of things I'm perfectly happy to let AI handle. Boilerplate, repetitive changes, things I've already done a hundred times, or just tedious work I don't feel like doing.

That's part of why I use these tools in the first place.

But when I'm building something unfamiliar, or working on functionality that I know I'll need to maintain later, I try to be a little more involved.

Sometimes that means just asking questions.

Other times I might just start changing things to see what happens.

I think even something as simple as predicting what a change will do before running it can make a difference. Because if the result isn't what I expected, now I have something to investigate.

Do I do this every time? Absolutely not.

Sometimes I've been staring at something for two hours, AI finally gets it working, and my entire thought process is basically "thank you, finally, moving on."

I'm not going to pretend otherwise.

But I am trying to be more conscious of when I'm actually learning something and when I'm just getting something done.

Because those aren't always the same thing.

Getting something working isn't the same as learning how it works

One of my favorite things about AI coding tools is how much more willing I am to try things I know very little about.

I've experimented with languages, frameworks, local models, desktop applications, and all kinds of other things that I probably would have put off before.

Not necessarily because I couldn't learn them, but because getting started on something completely unfamiliar can be a lot of work.

AI makes that initial hurdle so much smaller, and I think that's genuinely awesome.

But there's also a temptation to just keep building. One feature works, so you move on to the next one. Then the next. Before you know it, you've got a whole application and only a vague understanding of some pretty important parts of it.

And it's so easy to confuse all that progress with learning.

I've definitely done it.

I think the things that stick with me the most are usually the things I had to investigate a little. Something didn't behave the way I expected, or I got curious about why it worked a certain way, and I spent some time figuring it out.

Sometimes that investigation only takes a few minutes. Sometimes it sends me down a rabbit hole for the rest of the night.

Either way, I tend to remember those experiences a lot more than the features that worked perfectly on the first try.

And that's something I want to keep in mind as I continue building with AI.

I don't want to stop using these tools or intentionally make everything harder for myself.

I just don't want to get so caught up in how much I can build that I forget to pay attention to what I'm learning along the way.

So... what about you? Have you learned something useful while building with AI that you can now troubleshoot or explain yourself? Or have you ever gone back to AI-generated code and realized you had absolutely no idea how it worked?

Comments

[Pierre-Laurent Medori]

Your Slate example sent me to the repo, and the revision counter is right there in document.rs, pinned down by a test called edits_during_a_save_keep_the_document_dirty. A test is a nice way to remember the problem even after you've forgotten the code.

In the spirit of "the problem you didn't know existed", that same save path has one more: what's on disk if the app dies halfway through the write? write_text calls std::fs::write, which empties the file before writing, so a crash or power cut at the wrong moment leaves half a file. The funny part is that the fix is already in your repo: storage.rs has write_atomic (temp file, then rename) for the settings JSON. So right now your settings are better protected than your documents.

It's not a free swap for user files, though. Renaming over the original replaces it, so permissions, symlinks and hard links need care, and surviving a power cut also needs an fsync before the rename. That trade-off is why Vim has a whole option for it (backupcopy).

To answer your question: mine is a cousin of your save race. I once watched an API confirm a delete, then return the deleted item on the very next read. "The write succeeded" and "everyone can see it" turned out to be two different moments, and I've looked for that gap ever since.

[Jessica Doering]

Thank you so much for taking the time to actually dig through my repo. I genuinely appreciate comments like this. It’s one thing to have someone read a post, but when someone actually goes looking through the code and points out something I hadn’t considered, that’s incredibly valuable to me.

And you’re absolutely right, that’s another interesting edge case I hadn’t really thought about. Especially since I already have an atomic write implementation for the settings. Now you’ve given me something else to investigate.

I think that’s one of my favorite things about sharing projects publicly. I’m still learning, and getting feedback from people who take the time to actually understand what I’ve built is such a great part of that.

Also, your API delete example is a perfect example of what I was trying to get at in the post. It’s those little situations you run into that end up teaching you something you remember long after you’ve forgotten the actual code.

[Pierre-Laurent Medori]

Glad it was useful! Honestly, it works both ways. I'm always curious to see how other people tackle the problems I'm facing myself. It's enriching, and sometimes reassuring too, when someone else lands on the same idea I had.

[Fenix]

Caminante, son tus huellas
el camino y nada más;
Caminante, no hay camino,
se hace camino al andar.
Al andar se hace el camino,
y al volver la vista atrás
se ve la senda que nunca
se ha de volver a pisar.
Caminante no hay camino
sino estelas en la mar.

Todo pasa y todo queda
Pero lo nuestro es pasar
Pasar haciendo caminos
Caminos sobre la mar.

...es un extracto de un muy conocido poema de Antonio Machado.

[Fenix]

...nadie conoce cada esquina del mundo ni de la vida, y nadie sabe realmente hacia dónde va. ¡Y yo acabo de llegar! :) ...