My Machine Learning Project Taught Me More About Software Engineering Than Machine Learning

When I started building my ad click prediction project, I thought the hard part would be training a machine learning model.

I was wrong.

Before I even got to preprocessing the data, I found myself dealing with project architecture, testing, imports, packaging, and debugging environment issues. Looking back, those lessons were probably more valuable than the model itself.

Here's what happened.

It Started With a Notebook

Like many data science projects, everything began inside a Jupyter notebook.

Loading the dataset was as simple as:

df = pd.read_csv("data/raw/Advertising.csv")

It worked perfectly.

At first glance, there wasn't much reason to change it. The dataset loaded correctly, there were no missing values, no duplicates, and I could immediately start exploring the data.

But there was one problem.

The notebook knew how to load the data.

Nothing else did.

Separating Exploration From Reusable Code

Instead of leaving everything inside the notebook, I moved the loading logic into a dedicated module.

src/
└── data/
    └── load_data.py

The notebook changed from doing the work itself to simply calling a function:

from src.data.load_data import load_raw_data

df = load_raw_data("data/raw/Advertising.csv")

It doesn't feel like a huge improvement until you think about the future.

Now every notebook, script, or training pipeline can reuse the same code instead of copying and pasting it.

Small change.

Big difference.

Defensive Programming Matters

The next improvement wasn't about functionality.

It was about failure.

Instead of relying on pandas.read_csv() to throw an error if the dataset disappeared, I added an explicit check that raises a meaningful FileNotFoundError.

That seemed unnecessary at first.

But then I realized something.

Good software doesn't just work when everything is correct.

It also fails in ways that are easy to understand.

A descriptive error today can save a lot of debugging tomorrow.

Writing My First Test

The next step was introducing automated testing.

The first test was intentionally simple.

It verified that:

• the dataset loads successfully,
• it has the expected number of rows and columns,
• the target column exists.

Nothing fancy.

Still, it was the first time the project had a safety net.

Or so I thought.

The Test Didn't Fail Because of My Code

Instead, it failed with this:

ModuleNotFoundError: No module named 'src'

At first I assumed I had made a mistake.

I hadn't.

The notebook worked because I had manually modified sys.path.

pytest, however, starts a fresh Python process.

It had no idea where src lived.

The interesting part wasn't the error.

It was what the error revealed.

My project structure wasn't as solid as I thought.

The "Best Practice" Didn't Work

The first idea was to package the project properly using pyproject.toml and install it in editable mode.

That's the modern Python approach and, in many environments, it's absolutely the right solution.

Except...

My development environment had other ideas.

Running:

python -m pip install -e .

didn't work because of the packaging tools available in the hosted Anaconda environment.

For a moment, it felt like I'd gone down the wrong path.

But that turned into another lesson.

Sometimes the technically "best" solution isn't the practical one for your environment.

Pragmatism Beats Perfection

Instead of spending hours fighting the environment, I switched to a simpler approach.

Adding a small tests/conftest.py file allowed pytest to locate the project correctly.

One command later:

pytest

The result:

1 passed

Success.

More importantly, I learned something that applies far beyond Python.

Engineering isn't about blindly following best practices.

It's about choosing the right level of complexity for the problem you're solving.

The Real Milestone

By the end of this stage, I hadn't trained a model.

I hadn't engineered features.

I hadn't even started preprocessing.

Yet the project had already changed dramatically.

Instead of a single notebook, I now had:

• a reusable data loading module,
• a notebook that consumes reusable code instead of owning it,
• automated tests,
• a working pytest configuration,
• clearer error handling,
• a cleaner project structure.

That feels much closer to a production codebase than a collection of experiments.

What I'll Do Next

The next stage is building a reusable preprocessing pipeline.

Rather than cleaning data directly inside the notebook, I'll move that logic into src/data/preprocess.py, where every transformation can be tested and reused.

The notebook will become a client of the pipeline instead of the pipeline itself.

I'm beginning to appreciate that building a machine learning project isn't just about choosing algorithms.

It's about building software that happens to use machine learning.

And honestly, that's turning out to be the most valuable lesson so far.