I Started Learning Machine Learning After 8+ Years in IT — Here's What Building My First Churn Model Taught Me 🤖

gaurav singh — Sun, 16 Aug 2026 06:41:43 +0000

After spending 8+ years working in IT, I recently did something that felt surprisingly uncomfortable:

I became a beginner again.

Not because I had to.

Because I wanted to.

With AI changing the way software is built and consumed, I kept asking myself:

“What should I learn next?”

I could have watched videos, completed a course, collected a certificate and moved on.

Instead, I decided to build something.

That decision led me to my first end-to-end Customer Churn Prediction project.

And honestly, the biggest lessons weren't about which ML algorithm performed best.

They were about data, leakage, feature engineering, evaluation, interpretability, and the reality of learning something technical while working full-time.

Why Start ML After 8+ Years?

When you're a student, learning is your primary responsibility.

When you're working, learning happens somewhere between:

A full workday
Meetings
Production issues
Personal commitments
Family time
And whatever energy is left at the end of the day 😅

That's what made this journey interesting.

I wasn't learning Machine Learning for eight hours a day.

Most of the learning happened after work and on weekends.

Some days I was motivated.

Some days I was exhausted.

And some days I spent far too long trying to understand something that looked incredibly simple in a tutorial.

But there was one thing I kept reminding myself:

I don't need to learn everything. I just need to keep moving.

So I decided to stop consuming tutorials and start building.

🎯 The Problem: Customer Churn Prediction

The business problem was straightforward:

Can we predict which customers are at a higher risk of churning?

But I didn't want the project to stop at:

Input → Model → Churn / No Churn

I wanted to understand the complete ML workflow:

Raw Data
   ↓
Data Cleaning
   ↓
Feature Engineering
   ↓
Feature Selection
   ↓
Leakage Analysis
   ↓
Model Training
   ↓
Model Evaluation
   ↓
Feature Importance
   ↓
Business Insights

That turned out to be much more educational than simply training a classifier.

📊 78 Columns Looked Like a Good Starting Point

The initial dataset contained 78 columns.

My first instinct was:

“More features should give the model more information.”

But that's not necessarily true.

Some features were redundant.

Some didn't provide useful predictive information.

And some raised a much more serious question:

Would this information actually be available at the moment we need to make the prediction?

That question took me into one of the most important concepts I encountered during this project:

Data Leakage

A model can appear to perform extremely well if it gets access to information that would not actually be available when making a real-world prediction.

That's dangerous.

You can have a model with impressive validation metrics and still have a model that is practically useless.

For example, imagine predicting whether a customer will churn while accidentally including a feature that is only populated after the customer has already initiated the cancellation process.

The model isn't predicting churn.

It's recognizing evidence of churn that has already happened.

That distinction matters.

⏳ Look-Ahead Bias Was Another Wake-Up Call

The same idea becomes even more important when working with time-dependent data.

If you're predicting something at time T, your model should not have access to information from T+1.

Conceptually:

Past ---------------------> Prediction Time
          ↑
     Information
      available
          │
          X
     Future information
     must not leak back

This sounds obvious.

But when you're working with a real dataset containing dozens of columns, timestamps, transactions and historical records, it's surprisingly easy to introduce this problem accidentally.

This was one of the biggest lessons from the project:

Always ask: “Would I actually know this information at prediction time?”

🔧 Feature Engineering: 78 → 30

After analyzing the available features, I reduced the training dataset from:

78 columns → 30 meaningful features

This wasn't simply about deleting columns because they had low importance.

I looked at things such as:

Redundant information
Highly correlated features
Potential leakage
Temporal availability
Business meaning
Predictive usefulness

The important lesson for me was:

Feature engineering isn't just a preprocessing step. It is part of understanding the problem.

You can't meaningfully engineer features if you don't understand what the data represents.

🌲 Model Experiments

Once the feature set was ready, I experimented with several tree-based approaches.

Random Forest

A useful baseline and a good way to understand ensemble decision trees.

XGBoost

A powerful gradient-boosting approach that performed well in my experiments.

LightGBM

Another gradient-boosting implementation that I included in the comparison.

Stacking Ensemble

Finally, I experimented with combining models rather than relying on a single estimator.

The goal wasn't simply to find the model with the highest number.

I wanted to understand how different approaches behaved on the same feature set.

📏 Accuracy Wasn't Enough

One of the easiest mistakes when starting ML is to focus heavily on accuracy.

For churn prediction, that can be misleading.

Suppose:

95% of customers stay
5% of customers churn

A model that predicts:

Everyone → Stay

would have:

95% accuracy

But it would completely fail at the actual business problem.

That's why I evaluated the models using multiple metrics, including:

ROC-AUC
Precision
Recall
F1-score

The important lesson wasn't memorizing these metrics.

It was understanding why each metric exists and when it matters.

💡 Then I Asked: “Why Did the Model Predict This?”

This became one of my favourite parts of the project.

Imagine a model tells us:

Customer X has a high probability of churning.

That's useful.

But a business stakeholder will naturally ask:

“Why?”

A prediction without context can be difficult to act upon.

So I explored Permutation Feature Importance to understand which features were contributing most strongly to the model's predictions.

This helped move the project from:

Prediction

towards:

Prediction + Explanation

And that distinction is important.

Machine Learning isn't only about making predictions.

In many real-world scenarios, we need to understand whether those predictions make sense.

🧠 The Most Important ML Lesson I Learned

Before starting this project, I thought the Machine Learning workflow was something like:

Get Data
   ↓
Train Model
   ↓
Check Accuracy
   ↓
Done

Now I see it more like this:

Understand the Problem
        ↓
Understand the Data
        ↓
Clean the Data
        ↓
Engineer Features
        ↓
Check for Leakage
        ↓
Split Data Correctly
        ↓
Train Multiple Models
        ↓
Evaluate Using Appropriate Metrics
        ↓
Interpret the Model
        ↓
Translate Results into Business Meaning

The algorithm is only one part of the solution.

🧑‍💻 What Learning After 8+ Years Actually Felt Like

This is the part I didn't expect when I started.

After years of professional experience, you're generally comfortable with your domain.

You know how to solve problems.

You know how to debug.

You know how to communicate.

You know how to work under pressure.

Then you start learning something completely new.

Suddenly, you're back to:

Why isn't this working?

What does this metric actually mean?

Why is my model performing differently?

Did I introduce leakage?

Why did changing this feature affect the result?

And sometimes:

Maybe I should just watch another tutorial. 😅

But I eventually realized something.

Being experienced doesn't mean you should already know everything.

It means you should be comfortable learning when you don't.

🤖 Why I Think This Matters in the AI Era

The IT industry is changing quickly.

AI is changing development workflows.

Automation is changing repetitive tasks.

Job descriptions are evolving.

And the skills companies value are changing along with them.

I don't think the answer is:

“Everyone should become an ML engineer.”

That's unrealistic.

The answer, at least for me, is:

Understand the technology. Experiment with it. Learn how it works. Find where it can create value in your domain.

That's why I started this journey.

I didn't want to simply hear about AI every day.

I wanted to understand at least some of what was happening underneath the surface.

🚀 Three Months In

I'm still very much at the beginning.

I haven't learned everything.

I haven't worked with every ML technique.

And I definitely don't consider myself an AI/ML expert.

But three months ago, I couldn't say:

“I've built an end-to-end ML project.”

Now I can.

And that matters to me.

Not because the project is perfect.

It's not.

But because it represents something more important:

I started.

I made time after work.

I learned concepts I didn't understand.

I worked through confusing data.

I questioned my assumptions.

I built.

I tested.

I made mistakes.

And I learned.

🌱 If You're Also Thinking About Learning AI/ML

Maybe you're a developer.

Maybe you're a tester.

Maybe you're a data analyst.

Maybe you're in support, infrastructure, project management, or another part of IT.

Maybe you've been working for 5 years.

Maybe 10.

Maybe 15.

And you've been thinking:

“I should probably start learning AI.”

But you keep postponing it because you're already busy.

I understand.

You don't need to spend eight hours a day learning.

Start small.

Spend an hour.

Build something.

Break it.

Fix it.

Read documentation.

Ask questions.

Repeat.

Don't worry about becoming an expert immediately.

Focus on becoming better than you were yesterday.

🎯 What I Want to Explore Next

This project gave me a foundation, but it also gave me a much longer list of things I want to understand.

I'm particularly interested in going deeper into:

Better feature engineering
Model evaluation
Explainable ML
Model deployment
ML pipelines
Monitoring
Experiment tracking
And eventually taking ML projects from notebooks toward production

There is a long way to go.

And that's okay.

Final Thought

The biggest thing I built in these three months wasn't the churn prediction model.

It was the habit of learning again.

After 8+ years in IT, becoming a beginner wasn't easy.

But it was worth it.

Because technology will keep changing.

The tools will change.

The frameworks will change.

The models will change.

The industry will change.

And hopefully, so will I.

Not because I'm afraid of the future.

Because I'm curious about it. 🚀

A note about the project

This project was built as part of my AI/ML learning journey using anonymized information. Company-specific data, business details and sensitive information have intentionally been omitted.

If you're also learning Machine Learning, I'd genuinely love to hear:

What are you building right now?

machinelearning #python #datascience #ai #xgboost #lightgbm #randomforest #featureengineering #learninginpublic #devjourney

DEV Community: gaurav singh