DEV Community: Jospin Ndagano

Basics of Preprocessing in Machine Learning and Deep Learning

Jospin Ndagano — Mon, 25 Aug 2025 20:35:47 +0000

You’re testing your model on the validation set and the results look great. But when you try it on new data, the performance drops badly. After hours of searching, you find the problem: duplicate samples in your data and wrong handling of categorical features. These are common mistakes when preprocessing is done poorly. And you’re not alone, many people in machine learning say “dirty data” is their biggest challenge.

So, what is preprocessing? It simply means preparing your raw data fixing missing values, normalizing numbers, removing duplicates, encoding categories, so the model can learn properly.

Did you know that more than 80% of the time in AI projects is spent on data preparation, not on building models?

In this article, I’ll show you the key steps for good preprocessing, to save you from these problems and help your models perform better.

Different types of data require different preprocessing methods:

Numerical Data: Continuous values like age, salary, or temperature. Scaling and normalization are often necessary.

from sklearn.preprocessing import MinMaxScaler
X = [[20],[50],[80]]
print(MinMaxScaler().fit_transform(X))

Categorical Data: Discrete values like gender, country, or product category. Encoding is required.

from sklearn.preprocessing import LabelEncoder
labels = ["male","female","male"]
print(LabelEncoder().fit_transform(labels))

Text Data: Articles, reviews, or tweets. Requires tokenization, cleaning, and vectorization.

from sklearn.feature_extraction.text import CountVectorizer
texts = ["AI is cool","AI is powerful"]
print(CountVectorizer().fit_transform(texts).toarray())

Images and Audio: Pixels, spectrograms, or raw audio signals. Need resizing, normalization, and augmentation.

import numpy as np
img = np.array([[0,255],[128,64]])
print(img/255.0)  # normalize to [0,1]

Common Preprocessing Steps

a) Data Cleaning

Handling missing values (imputation or removal).

import pandas as pd
df = pd.DataFrame({"age":[20, None, 30]})
df["age"] = df["age"].fillna(df["age"].mean())  # imputation
print(df)

Detecting and removing outliers.

import numpy as np
data = np.array([10,12,11,500])  
clean = data[data < 100]  
print(clean)

Correcting errors and inconsistencies.

df = pd.DataFrame({"gender":["Male","male","Female"]})
df["gender"] = df["gender"].str.lower()
print(df)

b) Data Transformation

Normalization and Standardization: Ensures features are on the same scale.

df = pd.DataFrame({"gender":["Male","male","Female"]})
df["gender"] = df["gender"].str.lower()
print(df)

Encoding Categorical Variables: One-hot, label encoding, or embedding techniques.

from sklearn.preprocessing import OneHotEncoder
X = [["red"],["blue"],["red"]]
print(OneHotEncoder().fit_transform(X).toarray())

Date and Time Features: Extracting useful components like day, month, year, or weekday.

dates = pd.to_datetime(["2025-08-25","2025-08-26"])
print(dates.dt.day, dates.dt.weekday)

c) Feature Engineering

Creating new meaningful features.

df = pd.DataFrame({"length":[2,4],"width":[3,5]})
df["area"] = df["length"] * df["width"]
print(df)

Selecting the most important features.

from sklearn.feature_selection import SelectKBest, f_classif
import numpy as np
X = np.array([[1,10,20],[2,20,30],[3,30,40]])
y = [0,1,0]
print(SelectKBest(f_classif, k=2).fit_transform(X,y))

Dimensionality reduction techniques like PCA or t-SNE.

from sklearn.decomposition import PCA
X = [[1,2,3],[4,5,6],[7,8,9]]
print(PCA(2).fit_transform(X))

d) Deep Learning Specific Preprocessing

Images: Resizing, normalization, data augmentation (flip, rotate, crop).

import numpy as np
img = np.random.randint(0,255,(64,64))  
img_norm = img/255.0  # normalize
print(img_norm.shape)

Text: Tokenization, stemming/lemmatization, converting words to embeddings.

sentence = "AI is powerful"
tokens = sentence.lower().split()
vocab= {word: idx+1 for idx, word in enumerate(tokens)}
print(vocab)

Audio: Converting to spectrograms or MFCCs, normalizing amplitudes.

import librosa
y, sr = librosa.load(librosa.example('trumpet'))
mfccs = librosa.feature.mfcc(y=y, sr=sr)
print(mfccs.shape)

Overall, strong models start with clean data. By focusing on preprocessing, you unlock better accuracy, faster training, and reliable results.

References

Techtarget: What is data preprocessing? Key steps and techniques

Precisely: 2025 Planning Insights: Data Quality Remains the Top Data Integrity Challenge and Priority

Using Action Cable for Real-Time Applications in Ruby on Rails

Jospin Ndagano — Mon, 12 Aug 2024 15:01:48 +0000

Action Cable is a powerful feature in Ruby on Rails that allows developers to create real-time applications. It integrates WebSockets seamlessly into Rails, enabling bi-directional communication between the server and clients. This article will guide you through the basics of Action Cable and how to implement it in your Rails applications.

What is Action Cable?

Action Cable provides a framework for implementing WebSockets in Rails applications. WebSockets allow for persistent connections, enabling real-time features like chat applications, live notifications, and updates without requiring the client to refresh the page.

Setting Up Action Cable

To get started with Action Cable in your Rails application, follow these steps:

Create a New Rails Application If you don't have a Rails application set up yet, create one:

rails new my_realtime_app

cd my_realtime_app

Generate a Channel

Channels are the core components of Action Cable. They handle the WebSocket connections and define how data is sent and received.
Generate a new channel using the Rails generator:

rails generate channel Chat

This command creates several files, including:
app/channels/chat_channel.rb: The channel logic.
app/javascript/channels/chat_channel.js: The client-side JavaScript for the channel.

Implement the Channel Logic

Open app/channels/chat_channel.rb and define the behavior of the channel:

# app/channels/chat_channel.rb
class ChatChannel < ApplicationCable::Channel
  def subscribed
    stream_from "chat_channel" # Stream from this channel
  end

  def unsubscribed
    # Any cleanup needed when channel is unsubscribed
  end

  def receive(data)
    ActionCable.server.broadcast("chat_channel", message: data['message'])
  end
end

Set Up the Client-Side Code

Next, open app/javascript/channels/chat_channel.js and set up the client-side subscription:

// app/javascript/channels/chat_channel.js
import consumer from "./consumer";

const chatChannel = consumer.subscriptions.create("ChatChannel", {
  connected() {
    console.log("Connected to the chat channel!");
  },

  disconnected() {
    console.log("Disconnected from the chat channel.");
  },

  received(data) {
    // Handle incoming messages
    const messageElement = document.createElement("div");
    messageElement.innerText = data.message;
    document.getElementById("messages").appendChild(messageElement);
  },

  sendMessage(message) {
    this.perform("receive", { message: message });
  }
});

export default chatChannel;

Create a View for Chat

Now, create a simple view to display and send messages. Open app/views/chat/index.html.erb and add the following code:

<h1>Chat Room</h1>
<div id="messages"></div>
<input type="text" id="message_input" placeholder="Type your message here..." />
<button id="send_message">Send</button>

<script>
  document.addEventListener("DOMContentLoaded", () => {
    const sendMessageButton = document.getElementById("send_message");
    const messageInput = document.getElementById("message_input");

    sendMessageButton.addEventListener("click", () => {
      const message = messageInput.value;
      chatChannel.sendMessage(message);
      messageInput.value = ""; // Clear input field
    });
  });
</script>

Set Up the Routes

Add a route for the chat page in config/routes.rb:

Rails.application.routes.draw do
  root "chat#index"
end

Start the Rails Server

Run your Rails server:

rails server

Test the Application

Open your browser and navigate to http://localhost:3000. Open multiple tabs or windows to simulate different users in the chat. When you send a message from one window, it should appear in real-time in the other windows.

Action Cable makes it easy to add real-time features to your Ruby on Rails applications. By leveraging WebSockets, you can create interactive applications that respond instantly to user actions. Whether you're building a chat application, live notifications, or collaborative tools, Action Cable provides the tools you need to enhance user experience with real-time capabilities. Start exploring Action Cable in your projects to unlock the full potential of real-time web applications.

Test Your JavaScript App With Jest

Jospin Ndagano — Mon, 12 Aug 2024 14:40:42 +0000

Testing is a crucial part of software development that ensures your application behaves as expected. One of the most popular testing frameworks in the JavaScript ecosystem is Jest. Developed by Facebook, Jest is known for its simplicity and ease of use, making it an excellent choice for both beginners and experienced developers.

Getting Started with Jest

Before we dive into examples, let's set up Jest in your project. If you haven't already, you can install it using npm:

npm install --save-dev jest

Next, add a test script to your package.json:

"scripts": {
  "test": "jest"
}

Now you're ready to write some tests!
Let's create a simple function to test. Create a file named math.js:

// math.js
function add(a, b) {
  return a + b;
}
module.exports = add;

Now, let's write a test for this function. Create a file named math.test.js:

// math.test.js
const add = require('./math');

test('adds 1 + 2 to equal 3', () => {
  expect(add(1, 2)).toBe(3);
});

In this test, we use the test function to define a test case. The expect function is used to assert that the result of add(1, 2) is equal to 3.

Running the Tests

You can run your tests by executing the following command in your terminal:

npm test

You should see output indicating that your test has passed.

Testing Asynchronous Code

Jest also supports testing asynchronous code. Let's modify our math.js file to include an asynchronous function:

// math.js
function fetchData() {
  return new Promise((resolve) => {
    setTimeout(() => {
      resolve('peanut butter');
    }, 1000);
  });
}

module.exports = { add, fetchData };

Now, let's write a test for fetchData:

// math.test.js
const { add, fetchData } = require('./math');

test('adds 1 + 2 to equal 3', () => {
  expect(add(1, 2)).toBe(3);
});

test('fetches data', async () => {
  const data = await fetchData();
  expect(data).toBe('peanut butter');
});

In this test, we mark the test function as async and use await to wait for the promise to resolve.

Mocking Functions

Jest provides powerful mocking capabilities. Let's say you have a function that makes an API call. You can mock this function to control its behavior during tests.
Here's an example:

// api.js
const axios = require('axios');

async function getUser() {
  const response = await axios.get('https://api.example.com/user');
  return response.data;
}

module.exports = getUser;

You can mock axios in your test:

// api.test.js
const axios = require('axios');
const getUser = require('./api');

jest.mock('axios');

test('fetches user data', async () => {
  const user = { name: 'John Doe' };
  axios.get.mockResolvedValue({ data: user });

  const result = await getUser();
  expect(result).toEqual(user);
});

In this test, we mock the axios.get method to return a predefined response, allowing us to test getUser without making an actual API call.

Jest is a powerful testing framework that makes it easy to write and run tests for your JavaScript applications. With its simple syntax, support for asynchronous code, and built-in mocking capabilities, Jest can help you ensure your code is reliable and maintainable. Start integrating Jest into your projects today, and enjoy the benefits of automated testing.

Using RuboCop for Code Quality and Style Enforcement in Rails Projects

Jospin Ndagano — Mon, 12 Aug 2024 14:16:21 +0000

Maintaining code quality and consistency is crucial in any software development project, especially in Ruby on Rails applications. RuboCop is a popular static code analysis tool that helps enforce coding standards and improve the overall quality of your code. This article will guide you through using RuboCop in your Rails projects.

What is RuboCop?
RuboCop is a Ruby gem that provides a collection of style guidelines and best practices for Ruby code. It analyzes your code against these guidelines and suggests improvements, helping you catch potential issues early in the development process.

Setting Up RuboCop
To get started with RuboCop in your Rails project, follow these steps:

1. Add RuboCop to Your Gemfile
Open your Gemfile and add the following line:

group :development, :test do
  gem 'rubocop', require: false
end

After adding the gem, run the following command to install it:

bundle install

2. Create a RuboCop Configuration File
You can customize RuboCop's behavior by creating a configuration file. Run the following command to generate a default configuration file:

bundle exec rubocop --init

This command creates a .rubocop.yml file in the root of your project, where you can specify your custom rules and settings.
3. Configure RuboCop
Open the .rubocop.yml file and customize it according to your project's needs. Here’s a basic example:

# .rubocop.yml
AllCops:
  TargetRubyVersion: 2.7 # Specify your Ruby version
  Exclude:
    - 'db/schema.rb' # Exclude files or directories if needed

Metrics/LineLength:
  Max: 120 # Set maximum line length

Layout/IndentationWidth:
  Width: 2 # Set indentation width

4. Running RuboCop
To analyze your code, run the following command:

bundle exec rubocop

RuboCop will scan your codebase and provide a report of any offenses it finds, including style violations and potential issues.
5. Fixing Offenses
RuboCop often provides suggestions for fixing offenses. You can manually edit your code based on the feedback, or you can use the auto-correct feature to fix some issues automatically:

bundle exec rubocop -a

The -a flag tells RuboCop to automatically correct offenses when possible.
6. Integrating RuboCop with Your Editor
To enhance your development experience, consider integrating RuboCop with your text editor or IDE. Most popular editors, such as VSCode, RubyMine, and Atom, have plugins that provide real-time feedback as you write code.

Continuous Integration
To ensure code quality across your team, integrate RuboCop into your CI/CD pipeline. Add a step in your configuration (e.g., GitHub Actions, CircleCI) to run RuboCop during the build process. This way, any code that does not meet the specified standards will fail the build, promoting consistent quality.

Example GitHub Actions Configuration
Here’s a simple example of a GitHub Actions workflow that includes RuboCop:

name: Ruby on Rails CI

on: [push, pull_request]

jobs:
  rubocop:
    runs-on: ubuntu-latest
    steps:
      - name: Check out code
        uses: actions/checkout@v2
      - name: Set up Ruby
        uses: ruby/setup-ruby@v1
        with:
          ruby-version: '2.7'
      - name: Install dependencies
        run: |
          bundle install
      - name: Run RuboCop
        run: |
          bundle exec rubocop

Using RuboCop in your Rails projects is an effective way to enforce coding standards and improve code quality. By integrating RuboCop into your development workflow, you can catch potential issues early, maintain consistency across your codebase, and foster best practices among your team. Start using RuboCop today to elevate the quality of your Rails applications.

Branch and Bound Algorithm to Solve the Traveling Salesman Problem

Jospin Ndagano — Thu, 13 Jun 2024 09:05:57 +0000

In this blog post, we are going to take a look at how to use the Branch and Bound algorithm to solve the Traveling Salesman Problem and how to implement it using ruby.

First what is Branch and Bound algorithm? according to geeksforgeeks The Branch and Bound Algorithm is a method used in combinatorial optimization problems to systematically search for the best solution. It works by dividing the problem into smaller subproblems, or branches, and then eliminating certain branches based on bounds on the optimal solution. This process continues until the best solution is found or all branches have been explored.

Branch and Bound is commonly used to solve many problems like the Assignment Problem, Vehicle Routing Problem, Traveling Salesman Problem, Telecommunications Network Design, etc. But in this article we will use the Branch and Bound algorithm to solve the Traveling Salesman Problem and we'll implement it using ruby.

Before we dive deeper into the details let me give you an insight about what is the Traveling Salesman Problem.

Wikipedia says that the travelling salesman problem, also known as the travelling salesperson problem (TSP), asks the following question: "Given a list of cities and the distances between each pair of cities, what is the shortest possible route that visits each city exactly once and returns to the origin city?" It is an NP-hard problem in combinatorial optimization, important in theoretical computer science and operations research.

As said Wikipedia we'll give a list of cities and the distances between each pair of cities, the task is to find the shortest possible route that visits each city exactly once and returns to the origin city.

In other words, the goal is to determine the most efficient route for a salesman to visit a set of cities and return to the starting point, minimizing the total distance traveled.

Analyzing the above problem here are the steps we want to follow to solve the problem:

1. Initialization:

Represent the problem as a weighted graph G=(V,E), where 𝑉 is the set of cities and 𝐸 is the set of edges (roads) connecting the cities.
Define an initial solution (usually a random tour) and calculate its total distance.
Create a priority queue (or any other suitable data structure) to store the partial solutions.

require 'priority_queue'

class TravellingSalesmanProblem
  def initialize(cities)
    @cities = cities
    @graph = build_graph(cities)
    @best_tour = nil
    @best_distance = Float::INFINITY
    @priority_queue = PriorityQueue.new
    @priority_queue.push([], 0)
  end

  private

  def build_graph(cities)
    graph = {}
    cities.each do |city|
      graph[city] = cities.reject { |c| c == city }.map { |c| [c, distance(city, c)] }.to_h
    end
    graph
  end

  def distance(city1, city2)
    # Calculate the distance between two cities
    # (e.g., using Euclidean distance)
    Math.sqrt((city1[0] - city2[0])**2 + (city1[1] - city2[1])**2)
  end
end

2. Lower Bound Calculation:

For each partial solution in the priority queue, calculate a lower bound on the total distance of the complete tour.
One common way to calculate the lower bound is to find the minimum spanning tree of the remaining unvisited cities and add the cost of the minimum spanning tree to the current partial tour's distance.

def lower_bound(tour)
  unvisited_cities = @cities - tour
  return Float::INFINITY if unvisited_cities.empty?

  total_distance = tour.sum { |city| @graph[city].values.min }
  unvisited_cities.each do |city|
    total_distance += @graph[city].values.min
  end
  total_distance
end

3. Branching:

Select the partial solution with the smallest lower bound from the priority queue.
For each unvisited city in the selected partial solution, create a new partial solution by adding the city to the tour.
Calculate the lower bound for each new partial solution.

def branch_and_bound
  until @priority_queue.empty?
    tour, distance = @priority_queue.pop
    return tour if tour.length == @cities.length

    unvisited_cities = @cities - tour
    unvisited_cities.each do |city|
      new_tour = tour + [city]
      new_distance = distance + @graph[tour.last][city]
      new_lower_bound = lower_bound(new_tour)
      @priority_queue.push(new_tour, new_lower_bound)
    end
  end
  @best_tour
end

4. Pruning:

Compare the lower bound of each new partial solution with the current best solution's total distance.
If the lower bound of a partial solution is greater than or equal to the current best solution's total distance, discard that partial solution (i.e., prune the branch).

5. Update the Best Solution:

If a new partial solution's total distance is less than the current best solution's total distance, update the best solution.

def branch_and_bound
  until @priority_queue.empty?
    tour, distance = @priority_queue.pop
    return tour if tour.length == @cities.length

    if distance < @best_distance
      @best_tour = tour
      @best_distance = distance
    end

    # ... rest of the branch_and_bound method
  end
  @best_tour
end

6. Repeat:

Repeat steps 2 to 5 until the priority queue is empty or a stopping criterion is met (e.g., a time limit, a maximum number of iterations, or the optimal solution is found).

7. Output the Result:

The final best solution represents the optimal Traveling Salesman tour.

cities = [[0, 0], [1, 1], [2, 0], [3, 1]]
tsp = TravellingSalesmanProblem.new(cities)
optimal_tour = tsp.branch_and_bound
puts "Optimal tour: #{optimal_tour}"
puts "Optimal distance: #{tsp.best_distance}"

Here's the complete code:

require 'priority_queue'

class TravellingSalesmanProblem
  def initialize(cities)
    @cities = cities
    @graph = build_graph(cities)
    @best_tour = nil
    @best_distance = Float::INFINITY
    @priority_queue = PriorityQueue.new
    @priority_queue.push([], 0)
  end

  def branch_and_bound
    until @priority_queue.empty?
      tour, distance = @priority_queue.pop
      return tour if tour.length == @cities.length

      if distance < @best_distance
        @best_tour = tour
        @best_distance = distance
      end

      unvisited_cities = @cities - tour
      unvisited_cities.each do |city|
        new_tour = tour + [city]
        new_distance = distance + @graph[tour.last][city]
        new_lower_bound = lower_bound(new_tour)
        @priority_queue.push(new_tour, new_lower_bound)
      end
    end
    @best_tour
  end

  private

  def build_graph(cities)
    graph = {}
    cities.each do |city|
      graph[city] = cities.reject { |c| c == city }.map { |c| [c, distance(city, c)] }.to_h
    end
    graph
  end

  def distance(city1, city2)
    # Calculate the distance between two cities
    # (e.g., using Euclidean distance)
    Math.sqrt((city1[0] - city2[0])**2 + (city1[1] - city2[1])**2)
  end

  def lower_bound(tour)
    unvisited_cities = @cities - tour
    return Float::INFINITY if unvisited_cities.empty?

    total_distance = tour.sum { |city| @graph[city].values.min }
    unvisited_cities.each do |city|
      total_distance += @graph[city].values.min
    end
    total_distance
  end
end

cities = [[0, 0], [1, 1], [2, 0], [3, 1]]
tsp = TravellingSalesmanProblem.new(cities)
optimal_tour = tsp.branch_and_bound
puts "Optimal tour: #{optimal_tour}"
puts "Optimal distance: #{tsp.best_distance}"

This implementation uses the priority_queue gem to manage the partial solutions.

React rendering

Jospin Ndagano — Tue, 11 Jun 2024 14:22:47 +0000

Understanding the power of react rendering: Unlocking Efficient and Responsive User Experiences.

In the ever-evolving landscape of the web development, React has emerged as a dominant force, revolutionizing the way we build user interfaces. At the heart of React's success lies its efficient and responsive rendering process, which enables developers to create dynamic and high-performing applications.

The traditional approach to web development often involved directly manipulating the Document Object Model (DOM), the browser's representation of a web page. This process could be cumbersome and inefficient, as even minor changes to the UI would require the entire page to be re-rendered, resulting in sluggish performance and a sub optimal user experience.

React's innovative rendering process addresses these challenges head-on. Instead of directly updating the actual DOM, React introduces the concept of a virtual DOM, a lightweight in-memory representation of the UI. When a component's state changes, React compares the new virtual DOM with the previous one, identifying the specific differences. This process, known as "diffing" allows React to determine the minimal set of changes required to update the actual DOM, resulting in a more efficient responsive user experience.

The React rendering process can be broken down into the Following key steps:

1. Virtual DOM Creation: When a React component is first rendered, the Library creates a virtual DOM, a JavaScript object that mirrors the structure of the actual DOM. This virtual DOM serves as a lightweight and efficient representation of the UI.

2. State Changes: As the user interacts with the application's state changes, React updates the virtual DOM to reflect the new UI.

3. Diffing: React then compares the new virtual DOM with the previous version, identifying the specific differences between the two.

4. DOM Updates: Based on the identified differences, React update only the necessary parts of the actual DOM, rather than re-rendering the entire page. This process is known as "reconciliation."

By updating only the necessary parts of the DOM, React can significantly improve the performance of the application. Traditional approaches often required re-rendering the entire page, even for minor changes, leading to slow and sluggish user experiences.

React's efficient rendering process, however, minimizes the number of DOM operations required, resulting in faster and more responsive applications.

Another key benefit of React's rendering process is its cross-platform compatibility. The virtual DOM abstraction allows React to be used on various platform, such as web browsers, mobile devices, and even servers, without the need for significant changes to the rendering process. This flexibility enables developers to write code once deploy it across multiple environments, improving development efficiency and reducing maintenance overhead.

React's rendering process simplifies the development workflow. Developers can focus on updating the application's state, leaving the complex DOM manipulation tasks to the Library. This separation of concerns allows developers to create more modular and maintainable code, leading to improved application quality.

In conclusion, the power of React's rendering process lies in its ability to efficiently update the DOM, providing users with a smooth and responsive experience. By leveraging the virtual DOM and the diffing algorithm, React can minimize the number of DOM operations required, resulting in faster and more efficient applications. This rendering approach, combined with React's cross-platform compatibility and simplified development workflow, has made it a go-to choice for building modern, high-performance user interfaces. As the web development landscape continues to evolve, React's rendering process will undoubtedly remain a key factor in the creation of exceptional user experiences.