DEV Community: Bee

Why Data Cleaning is 80% of Data Science

Bee — Thu, 21 Aug 2025 10:29:53 +0000

When most people imagine Data Science, they picture building shiny machine learning models or crafting beautiful dashboards.
But here’s the truth: if your data is messy, no model can save you.

That’s why people say:
👉 “80% of a data scientist’s time is spent cleaning data.”

And it’s not a joke.

Common Data Cleaning Nightmares

Missing values → blank cells, NaNs, or inconsistent entries.
Duplicates → one user recorded multiple times.
Outliers → extreme values like a salary of $1,000,000,000.
Inconsistent categories → Nairobi, NBO, 254-Nairobi treated as different.
Wrong data types → numbers stored as text.

🛠 Practical Cleaning in Python (with Pandas)

Here’s how I usually tackle some of these:

import pandas as pd

# Example dataset
data = {
    "Name": ["Alice", "Bob", "Charlie", "Alice"],
    "Age": [25, None, 37, 25],
    "Salary": [50000, 60000, 1000000000, 50000],  # notice the outlier
    "City": ["Nairobi", "NBI", "254-Nairobi", "Nairobi"]
}

df = pd.DataFrame(data)

print("Before Cleaning:")
print(df)

# 1. Handle missing values
df["Age"].fillna(df["Age"].median(), inplace=True)

# 2. Drop duplicates
df.drop_duplicates(inplace=True)

# 3. Fix inconsistent categories
df["City"] = df["City"].replace({"NBI": "Nairobi", "254-Nairobi": "Nairobi"})

# 4. Handle outliers (example: cap salaries above 200k)
df["Salary"] = df["Salary"].apply(lambda x: min(x, 200000))

print("\nAfter Cleaning:")
print(df)

✅ Output

Before Cleaning:

      Name   Age       Salary         City
0    Alice  25.0       50000      Nairobi
1      Bob   NaN       60000          NBI
2  Charlie  37.0  1000000000  254-Nairobi
3    Alice  25.0       50000      Nairobi

After Cleaning:

      Name   Age  Salary     City
0    Alice  25.0   50000  Nairobi
1      Bob  25.0   60000  Nairobi
2  Charlie  37.0  200000  Nairobi

Key Takeaways

Don’t ignore missing values, understand the why before filling/dropping.
Standardize categories early to avoid “ghost” groups.
Outliers may be errors or real rare events, investigate before deleting.
A clean dataset can outperform a complex model trained on dirty data.

django architecture;MVC vs MVT

Bee — Tue, 01 Jul 2025 19:27:41 +0000

In this post, we'll demystify both patterns and show how Django's MVT is related to the classic MVC. Let’s get into it!

MVC

The MVC (Model-View-Controller) design pattern is a software architectural pattern that separates application logic into three interconnected components:

Model: The part that handles the data. It defines how data is stored, retrieved, and manipulated — usually tied to a database.
View: The UI or representation layer. It presents the data to the user.
Controller: The traffic cop that handles user input, updates the model, and decides which view to show.

This separation makes applications easier to scale and maintain.

Here's a visual breakdown:

MVC is widely used in frameworks like Ruby on Rails, Laravel (PHP), and ASP.NET.

Enter Django: The MVT Way

Django follows the MVT (Model-View-Template) architectural pattern, which is a variation of the traditional MVC (Model-View-Controller) design pattern used in web development. This pattern separates the application into three main components:

Model (same as MVC): Manages the data — built using Django’s ORM. Defines the structure of your database.
View (different from MVC): In Django, the “View” contains the business logic. It fetches data from the model and passes it to the template.
Template: Responsible for rendering the final HTML — your front-end content.

Now here’s the diffrence:

In Django, the "View" from MVC is called the "Template", and the "Controller" role is handled by the Django framework itself.

So Django's View is actually the Controller in traditional MVC!

here is the visual breakdown

🔁 Side-by-Side: MVC vs MVT

Concept	MVC	Django (MVT)
Model	Handles data & logic	Handles data & logic
View	UI layer	Template (HTML)
Controller	Handles user input	View (Python functions)
Framework	Passive	Active controller

here is the visual breakdown!

So Django automates a lot of what traditional MVC expects you to write manually.

🛠️ Example: A Simple Blog

Let’s say we’re building a blog:

Model:

# models.py
class Post(models.Model):
    title = models.CharField(max_length=200)
    content = models.TextField()
    date_posted = models.DateTimeField(auto_now_add=True)

View:

# views.py
from django.shortcuts import render
from .models import Post

def home(request):
    posts = Post.objects.all()
    return render(request, 'blog/home.html', {'posts': posts})

Template:

<!-- home.html -->
{% for post in posts %}
  <h2>{{ post.title }}</h2>
  <p>{{ post.content }}</p>
  <small>{{ post.date_posted }}</small>
{% endfor %}

This is the heart of Django’s MVT — clean separation, yet tightly integrated by Django’s robust request handling.

Why MVT Works So Well

Separation of concerns: You work on templates separately from the business logic and data models.
Built-in admin panel: The model definitions give you an auto-generated backend.
Scalability: The architecture supports large projects out-of-the-box.
Rapid development: You can go from idea to MVP in record time.

✨ Final Thoughts

By understanding how MVT maps to traditional MVC, you'll appreciate Django’s design even more. It's MVC with a twist — and that twist is what makes Django so developer-centric.

📝 References:

here are the links to learn more;

Django: my first django project

Bee — Mon, 30 Jun 2025 20:17:46 +0000

on day four of django learning i created a simple project following these steps

Pre-requisites

Python installed
Basic command-line knowledge install django on your terminal run

  pip install django

1. 📁 Start a Django Project

In your terminal:

django-admin startproject myproject
cd myproject
python manage.py startapp restaurant
python manage.py startapp shop

myproject is the project name
shop and restaurant are the names of the site i wanted to create

Now, my structure looks like this :

open shop and it will display this

admin.py: Configuration for the Django admin interface.
apps.py: Configuration for the app itself.
models.py: Contains the database models for the app.
tests.py: Contains tests for the app.
views.py: Contains the request/response logic for the app.
migrations/: Contains database migrations for the app

2. ⚙️ Register the App

Open myproject/settings.py, find INSTALLED_APPS, and add ' shop'and 'restaurant':

INSTALLED_APPS = [
    ...
    'shop',
    'restaurant',
]

this is how it looks!

3. 🧱 Create a shop Model

we shall start with shop model

In shop/models.py:

from django.db import models

class Post(models.Model):
    title = models.CharField(max_length=100)
    content = models.TextField()

    def __str__(self):
        return self.title

Then run:

python manage.py makemigrations
python manage.py migrate

4. 👑 Register Model in Admin

In shop/admin.py:

from django.contrib import admin
from .models import Post

admin.site.register(Post)

here is how it looks

Create a superuser:

python manage.py createsuperuser

Run the server:

python manage.py runserver

Go to http://127.0.0.1:8000/admin/ and add a few blog posts!

5. Show Posts in the Browser

In `shop/views.py`:

from django.shortcuts import render
from .models import Post

def home(request):
    posts = Post.objects.all()
    return render(request, 'shop/home.html', {'posts': posts})

Create a file: `shop/urls.py`

from django.urls import path
from . import views

urlpatterns = [
    path('', views.home, name='blog-home'),
]

Link it in your main `myproject/urls.py`:

from django.contrib import admin
from django.urls import path, include

urlpatterns = [
    path('admin/', admin.site.urls),
    path('shop/', include('shop.urls')),
]

6. 🖼 Create a Template

Create folders:

shop/templates/shop/home.html

Then add:

<!DOCTYPE html>
<html>
<head>
    <title>My shop</title>
</head>
<body>
    <h1>shop Posts</h1>
    {% for post in posts %}
        <h2>{{ post.title }}</h2>
        <p>{{ post.goods }}</p>
        <hr>
    {% endfor %}
</body>
</html>

🎉 Done!

Now visit:
👉 http://127.0.0.1:8000/shop/
And boom — your shop model will appear!

version control on git and github

Bee — Thu, 26 Jun 2025 10:39:14 +0000

in this article i will be sharing what i have learnt about

Version control
Forking
Collaboration
Pull Requests
Merge Conflicts
Code Reviews
GitHub Issues
Git Commands
Pushing Changes to GitHub

Let’s get into it!
(a)version control
Version control software helps facilitate continuous software development workflows. As user demands scale up, version control helps developers work smarter together, using time and resources more efficiently.
It keeps a historical record of software changes in a specialized database, logging edits made by individual developers. When conflicts emerge, developers can look back and resolve code conflicts, minimizing disruption to the codebase.

here is the article i used to learn
_https://github.com/resources/articles/software-development/what-is-version-control
_
(b) forking
forking is the process of creating your own copy of someone's else project on github ,it is mainly used if you dont have access to the original respiratory
If you have write access to the original repository, you don’t need a fork. Instead, use branches to manage your work. If you don’t have write access to a repository you want to contribute to, fork it. Make your changes in your fork, then submit them through a merge request to the upstream repository.

here is the article with more details
_https://docs.gitlab.com/user/project/repository/forking_workflow/
(c)collaboration
Collaboration in GitHub means working on shared code with others—whether you're on the same team or contributing to someone else’s project.

Here’s what good collaboration includes:

Keeping commits clean and meaningful
Creating branches for features or fixes
Communicating through issues, PRs, and comments

Pro tip: Branch before you make changes (git checkout -b feature-branch) and avoid working directly on the main branch.

(d) pull request(pr)
pull request (PR) is how you propose changes from your branch (or fork) into another repository.

📥 Basic Steps:

Fork or clone the repo
Make changes in a new branch
Push to your GitHub repo
Click “New pull request” in GitHub

It opens a discussion space where reviewers can comment, approve, or request changes.

here is the link to learn more
https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request

(e) merge conflicts
Merge conflicts happen when changes from different branches clash—often when two people edit the same lines of code.

How to resolve them:

Git tells you which files have conflicts.
Open the file and manually choose what to keep.
After editing, run: git add . git commit

here is the link to learn more
https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/addressing-merge-conflicts

(f) Code Review
Before merging a PR, team members usually **review **your code. This ensures quality, consistency, and avoids bugs.

In code review, I learned to:

Give clear commit messages
Write readable code
Leave helpful comments (when reviewing others)

(g) GitHub Issues
Issues are a great way to track bugs, feature requests, or general tasks in a project. I’ve used them to:

Plan what needs to be done
Assign tasks in teams
Link PRs to issues for better tracking
You can add labels, assign people, or even automate issue handling with bots.

(h)Useful Git Commands
**Here are some of the commands to use regularly:
git clone <repo-url> # Copy a repo to your machine git checkout -b branch-name # Create and switch to a new branch git status # Check what has changed git add . # Stage all changes git commit -m "message" # Save changes with a message git push # Upload changes to GitHub git pull # Get the latest changes from remote git fetch # Check for updates without merging git merge # Merge another branch into your current one
here is the link to git cheat sheet
_https://training.github.com/downloads/github-git-cheat-sheet/

_(i) Pushing Changes to GitHub
**Once the work is done locally, here’s how to push it to GitHub:

git push origin branch-name

Then open a pull request from GitHub’s UI. This step is key in contributing to projects and sharing your work with others.
(j)final thoughts
finally I would recommend using this site for practical experience
_https://skills.github.com/
_

# Set Up My Dev Environment: Git, Docker, and GitHub SSH on Linux

Bee — Thu, 26 Jun 2025 07:33:03 +0000

Hi everyone! 👋

I recently set up my development environment and thought I’d share how I installed Git, Docker, and configured Git with GitHub SSH on my machine. If you’re getting started with development or just want a clean setup, this guide might help you too!

🔧 1. Installing Git

Git is essential for version control, so it was my first step.

🖥️ Commands I Used:

sudo apt update
sudo apt install git

✅ Verifying Git Installation:

bash git --version

If installed correctly, this returns something like:

git version 2.xx.x

🐳 2. Installing Docker

Next up was Docker — a tool I’ll be using to containerize apps and run environments in isolated containers.

Steps I Followed:

Removed old Docker versions (just in case):

bash sudo apt remove docker docker-engine docker.io containerd runc

Installed dependencies:

bash sudo apt update sudo apt install ca-certificates curl gnupg

Added Docker’s GPG key:

bash sudo install -m 0755 -d /etc/apt/keyrings curl -fsSL https://download.docker.com/linux/ubuntu/gpg | \ sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg

Added the Docker repository:

bash echo \ "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \ https://download.docker.com/linux/ubuntu \ $(lsb_release -cs) stable" | \ sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

Installed Docker Engine:

bash sudo apt update sudo apt install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Verified Docker Installation:

bash docker --version

This gave me something like:

Docker version XX.XX.X, build XXXXX

(Optional) To use Docker without typing sudo every time:

bash sudo usermod -aG docker $USER newgrp docker

🔐 3. Configuring Git with GitHub via SSH

To avoid typing my GitHub credentials every time I push changes, I set up Git to authenticate using SSH.

🧑‍💻 Steps I Took:

Generated a new SSH key:

bash ssh-keygen -t ed25519 -C "myemail@example.com"

I accepted the default file location and passphrase.

Added the SSH key to the agent:

bash eval "$(ssh-agent -s)" ssh-add ~/.ssh/id_ed25519

Copied my SSH key:

bash cat ~/.ssh/id_ed25519.pub

Added the key to GitHub:

Went to GitHub > Settings > SSH and GPG keys
Clicked “New SSH Key”
Pasted the key and saved

Tested the SSH connection:

bash ssh -T git@github.com

It responded with:

Hi your-username! You've successfully authenticated...

Set up Git username and email globally:

bash git config --global user.name "Your Name" git config --global user.email "your-email@example.com"

✅ Final Thoughts

Now my environment is ready for development with:

📁 Git for version control
🐳 Docker for containerization
🔐 GitHub SSH for secure, password-free pushing and pulling

This setup will help me manage projects more efficiently and collaborate more smoothly. If you're setting up yours too and get stuck—feel free to drop a question!