DEV Community: Juma Shafara

DATAIDEA WALLPAPER

Juma Shafara — Thu, 05 Dec 2024 12:11:49 +0000

Descriptive Statistics

Juma Shafara — Sat, 24 Aug 2024 05:37:37 +0000

## Uncomment and run this cell to install the packages
# !pip install --upgrade dataidea

Descriptive Statistics and Summary Metrics

In this notebook, we will learn to obtain important values that describe our data including:

Measures of central tendency
Measures of variability
Measures of distribution shape
Measures of association

import pandas as pd
import numpy as np
import scipy as sp
import matplotlib.pyplot as plt

This notebook has been modified to use the Nobel Price Laureates Dataset which you can download from opendatasoft

# load the dataset (modify the path to point to your copy of the dataset)
data = pd.read_csv('../assets/nobel_prize_year.csv')
data = data[data.Gender != 'org'] # removing organizations
data.sample(n=5)

	Year	Gender	Category	birth_year	age
505	2014	male	Physics	1929	85
318	1952	male	Literature	1885	67
883	1933	male	Literature	1870	63
481	1995	male	Peace	1908	87
769	2005	male	Peace	1942	63

Don't Miss Any Updates!

Before we continue, I have a humble request, to be among the first to hear about future updates of the course materials, simply enter your email below, follow us on (formally Twitter), or subscribe to our YouTube channel.

What is Descriptive Statistics

Descriptive statistics is a branch of statistics that deals with the presentation and summary of data in a meaningful and informative way. Its primary goal is to describe and summarize the main features of a dataset.

Commonly used measures in descriptive statistics include:

Measures of central tendency: These describe the center or average of a dataset and include metrics like mean, median, and mode.
Measures of variability: These indicate the spread or dispersion of the data and include metrics like range, variance, and standard deviation.
Measures of distribution shape: These describe the distribution of data points and include metrics like skewness and kurtosis.
Measures of association: These quantify the relationship between variables and include correlation coefficients.

Descriptive statistics provide simple summaries about the sample and the observations that have been made.

1. Measures of central tendency ie Mean, Median, Mode:

The Center of the Data:

The center of the data is where most of the values are concentrated.

Mean: It is the average value of a dataset calculated by summing all values(numerical) and dividing by the total count.

mean_value = np.mean(data.age)
print("Mean:", mean_value)

Mean: 60.21383647798742

Median: It is the middle value of a dataset when arranged in ascending order. If there is an even number of observations, the median is the average of the two middle values.

median_value = np.median(data.age)
print("Median:", median_value)

Median: 60.0

Mode: It is the value that appears most frequently in a dataset.

mode_value = sp.stats.mode(data.age)[0]
print("Mode:", mode_value)

Mode: 56

Homework:

Other ways to find mode (ie using pandas and numpy)

2. Measures of variability

The Variation of the Data:

The variation of the data is how spread out the data are around the center.

a) Variance and Standard Deviation:

Variance: It measures the spread of the data points around the mean.

# how to implement the variance and standard deviation using numpy
variance_value = np.var(data.age)
print("Variance:", variance_value)

Variance: 159.28551085795658

Standard Deviation: It is the square root of the variance, providing a measure of the average distance between each data point and the mean.

std_deviation_value = np.std(data.age)
print("Standard Deviation:", std_deviation_value)

Standard Deviation: 12.620836377116873

Summary

In summary, variance provides a measure of dispersion in squared units, while standard deviation provides a measure of dispersion in the original units of the data

Note!

Smaller variances and standard deviation values mean that the data has values similar to each other and closer to the mean and the vice versa is true

plt.hist(x=data.age, bins=20, edgecolor='black')
# add standard deviation lines
plt.axvline(mean_value, color='red', linestyle='--', label='Mean')
plt.axvline(mean_value+std_deviation_value, color='orange', linestyle='--', label='1st std Dev')
plt.axvline(mean_value-std_deviation_value, color='orange', linestyle='--')
plt.title('Age of Nobel Prize Winners')
plt.ylabel('Frequency')
plt.xlabel('Age')
# Adjust the position of the legend
plt.legend(loc='upper left')

plt.show()

b) Range and Interquartile Range (IQR):

Range: It is the difference between the maximum and minimum values in a dataset. It is simplest measure of variation

# One way to obtain range
min_age = min(data.age)
max_age = max(data.age)
age_range = max_age - min_age
print('Range:', age_range)

Range: 80

# Calculating the range using numpy
range_value = np.ptp(data.age)
print("Range:", range_value)

Range: 80

Interquartile Range (IQR): It is the range between the first quartile (25th percentile) and the third quartile (75th percentile) of the dataset.

Quartiles:

Calculating Quartiles

The quartiles (Q0,Q1,Q2,Q3,Q4) are the values that separate each quarter.

Between Q0 and Q1 are the 25% lowest values in the data. Between Q1 and Q2 are the next 25%. And so on.

Q0 is the smallest value in the data.
Q1 is the value separating the first quarter from the second quarter of the data.
Q2 is the middle value (median), separating the bottom from the top half.
Q3 is the value separating the third quarter from the fourth quarter
Q4 is the largest value in the data.

# Calculate the quartile
quartiles = np.quantile(a=data.age, q=[0, 0.25, 0.5, 0.75, 1])

print('Quartiles:', quartiles)

Quartiles: [17. 51. 60. 69. 97.]

Percentiles:

Percentiles are values that separate the data into 100 equal parts.

For example, The 95th percentile separates the lowest 95% of the values from the top 5%

The 25th percentile (P25%) is the same as the first quartile (Q1).
The 50th percentile (P50%) is the same as the second quartile (Q2) and the median.
The 75th percentile (P75%) is the same as the third quartile (Q3)

Calculating Percentiles with Python

To get all the percentile values, we can use np.percentile() method and pass in the data, and the list of the percentiles as showed below.

# Getting many percentiles
percentiles = np.percentile(data.age, [25, 50, 75])
print(f'Percentiles: {percentiles}')

Percentiles: [51. 60. 69.]

To get a single percentile value, we can again use the np.percentile() method and pass in the data, and a the specicific percentile you're interested in eg:

# Getting one percentile at a time
first_quartile = np.percentile(a=data.age, q=25) # 25th percentile
middle_percentile = np.percentile(data.age, 50)
third_quartile = np.percentile(data.age, 75) # 75th percentile

print('Q1: ', first_quartile)
print('Q2: ', middle_percentile)  
print('Q3: ', third_quartile)

Q1:  51.0
Q2:  60.0
Q3:  69.0

Note!

Note also that we can be able to use the `np.quantile()` method to calculate the percentiles which makes logical sense as all the values mark a fraction(percentage) of the data

percentiles = np.quantile(a=data.age, q=[0.25, 0.50, 0.75])
print('Percentiles:', percentiles)

Percentiles: [51. 60. 69.]

Now we can be able to obtain the interquartile range as the difference between the third and first quartiles as predefined.

# obtain the interquartile
iqr_value = third_quartile - first_quartile
print('Interquartile range: ', iqr_value)

Interquartile range:  18.0

Note: Quartiles and percentiles are both types of quantiles

Summary

While the range gives an overview of the entire spread of the data from lowest to highest, the interquartile range focuses s`pecifically on the spread of the middle portion of the data, making it more robust against outliers.

Frequency Tables

Frequency means the number of times a value appears in the data. A table can quickly show us how many times each value appears. If the data has many different values, it is easier to use intervals of values to present them in a table.

Here's the age of the 934 Nobel Prize winners up until the year 2020. IN the table, each row is an age interval of 10 years

Age Interval	Frequency
10-19	1
20-29	2
30-39	48
40-49	158
50-59	236
60-69	262
70-79	174
80-89	50
90-99	3

Note: The intervals for the values are also called bin

What's on your mind? Put it in the comments!

Why Learning Programming Can Be Hard and How to Make It Easier

Juma Shafara — Mon, 12 Aug 2024 16:27:24 +0000

Learning to program is like picking up a new language—one that's not only spoken but also written and logically structured. For many, it's a journey filled with excitement, challenges, and, at times, frustration. But why is programming so difficult to learn, and how can you make the process smoother? Let's dive in.

Why Is Learning Programming So Hard?

Abstract Thinking: Programming is all about solving problems in a way that a computer can understand. This requires abstract thinking—something that's not typically used in day-to-day problem-solving. You're often asked to break down a problem into smaller parts, think about how data flows, and structure logic in a way that's both efficient and clear. This shift in thinking can be tough to get used to.
Syntax and Semantics: Every programming language has its own rules (syntax) and meanings (semantics). It's similar to learning the grammar and vocabulary of a new spoken language, but with even less room for error. A small mistake, like missing a semicolon, can lead to frustrating errors that prevent your code from running.
Debugging and Error Handling: Let's face it—no one writes perfect code on the first try. Debugging is a core part of programming, but it can be frustrating, especially when you're just starting out. Learning to systematically identify and fix errors requires patience and a methodical approach, which can be challenging to develop.
Rapid Technological Change: The tech world moves fast. New programming languages, frameworks, and tools are constantly emerging. While this is exciting, it can also make learning programming feel like chasing a moving target. Just when you think you've mastered one thing, another, seemingly better tool comes along.
Overwhelming Resources: There’s no shortage of learning resources out there, but this abundance can be overwhelming. With countless tutorials, courses, and books, it’s hard to know where to start and which path to follow. It's easy to get lost in the sea of information and lose sight of your learning goals.
Complex Concepts: Programming introduces concepts that can be difficult to grasp at first. Ideas like recursion, pointers, object-oriented programming, and concurrency are powerful, but they can also be complex and confusing, especially if you don't have a strong foundation in the basics.

How to Make Learning Programming Easier

Start Small: Don’t try to learn everything at once. Choose a programming language with a simple syntax, like Python, and focus on mastering the basics. Learn how to write simple programs before diving into more complex topics.
Practice Regularly: Consistency is key in learning programming. Aim to code every day, even if it's just for a short time. Regular practice helps reinforce what you’ve learned and makes it easier to progress.
Build Projects: Theory is important, but applying what you’ve learned is where the real understanding happens. Start with small projects that interest you—whether it’s a simple calculator or a basic website. Projects not only reinforce your learning but also make the process more enjoyable.
Break Down Problems: Programming often involves solving complex problems, but these can usually be broken down into smaller, more manageable parts. Start by tackling the simplest piece of the problem and gradually build up to the complete solution. This approach makes the task less overwhelming and more approachable.
Learn to Debug: Debugging is an essential skill for any programmer. When your code doesn’t work as expected (and it won’t, at first), don’t get discouraged. Instead, use it as a learning opportunity. Pay attention to error messages, use debugging tools, and try to understand why the error occurred.
Join a Community: You don’t have to learn programming alone. Join online communities, coding bootcamps, or local meetups where you can connect with other learners and experienced programmers. These communities provide support, motivation, and answers to your questions.
Choose Quality Resources: With so many resources available, it’s important to choose a few high-quality ones and stick with them. Look for reputable books, courses, or tutorials that provide a structured learning path and explain concepts clearly.
Learn by Teaching: One of the best ways to solidify your understanding of programming is by teaching it to others. Whether it’s writing blog posts, creating tutorials, or helping others in coding forums, teaching forces you to organize your thoughts and explain concepts clearly.
Stay Patient and Persistent: Learning programming takes time, and you’re bound to hit some roadblocks along the way. Don’t be discouraged by setbacks. Stay patient, keep practicing, and remember that every mistake is a step towards mastery.
Practice Problem-Solving: Engage in coding challenges and exercises that focus on problem-solving. Websites like LeetCode, HackerRank, and Codewars offer a variety of problems that can help you improve your skills and gain confidence in your coding abilities.

Final Thoughts

Learning to program is a journey—one that requires time, effort, and a lot of patience. But it’s also incredibly rewarding. By starting small, practicing regularly, building projects, and joining a community, you can overcome the challenges and find joy in coding. Remember, every great programmer started where you are now, so keep pushing forward. The effort you put in today will pay off tomorrow.

Happy coding!

Linux Mint, A Comprehensive Overview

Juma Shafara — Mon, 29 Jul 2024 13:20:54 +0000

If you're new to the world of Linux or looking for a robust, user-friendly operating system, Linux Mint might just be your perfect match. This popular Linux distribution has earned its reputation for being accessible, reliable, and packed with features. I recently started using Linux Mint and in this blog, we'll explore what makes Linux Mint special and why it might be the ideal choice for your next operating system.

A User-Friendly Interface

Cinnamon Desktop:
Linux Mint’s flagship edition features the Cinnamon desktop environment, designed to provide a familiar and intuitive user experience. If you’re coming from Windows, you’ll find the transition smooth and straightforward. The interface is clean, modern, and highly customizable, ensuring that both new and seasoned users feel right at home.

Alternative Desktops:
For those with different preferences or hardware requirements, Linux Mint also offers MATE and Xfce editions. MATE provides a classic desktop experience, while Xfce is lightweight and optimized for older or less powerful hardware.

Out-of-the-Box Experience

Pre-installed Software:
One of the standout features of Linux Mint is its comprehensive suite of pre-installed software. From office applications to multimedia tools and system utilities, Linux Mint comes ready to use right after installation. You won’t have to spend time hunting for basic applications, making your transition smoother.

Media Codecs:
Unlike some other Linux distributions, Linux Mint includes a wide range of multimedia codecs by default. This means you can play various audio and video formats out of the box, without needing to install additional codecs.

Stability and Performance

Based on Ubuntu LTS:
Linux Mint is built on top of Ubuntu’s Long Term Support (LTS) releases, providing a stable and well-supported foundation. This ensures you receive regular updates and security patches, keeping your system secure and up-to-date.

Optimized Performance:
Linux Mint is optimized for performance, making it suitable for both modern and older hardware. Whether you’re using the latest laptop or an aging desktop, Linux Mint ensures a smooth and responsive experience.

Ease of Use and Configuration

MintTools:
Linux Mint includes a set of custom tools designed to make system management and configuration a breeze. Tools like the Update Manager, Software Manager, and System Settings are intuitive and user-friendly, simplifying tasks like software installation, system updates, and hardware configuration.

Easy Installation:
The installation process is straightforward, with a user-friendly installer that guides you through partitioning and system setup. Even if you’re new to Linux, you’ll find the installation process hassle-free.

Community and Support

Active Community:
Linux Mint boasts an active and supportive community. Forums, documentation, and other resources are readily available to help you troubleshoot issues and learn more about your system. Whether you’re a beginner or an experienced user, you’ll find plenty of support within the Linux Mint community.

Regular Updates:
Linux Mint receives regular updates and new releases, ensuring you have access to the latest features and improvements. The development team is dedicated to providing a top-notch user experience, continually refining and enhancing the system.

Customization and Flexibility

Highly Customizable:
Linux Mint’s desktop environments are highly customizable, allowing you to tailor the look and feel of your system to your liking. From themes and icons to panel layouts and applets, the customization options are virtually endless.

Software Repositories:
With access to Ubuntu’s extensive software repositories as well as Mint’s own repository, you have a vast array of software at your fingertips. Installing new applications is easy and straightforward, giving you the flexibility to expand your system’s capabilities.

Security and Privacy

Enhanced Security:
Linux Mint places a strong emphasis on system security, providing regular security updates to protect you from vulnerabilities. The system is designed to be secure out of the box, giving you peace of mind.

Privacy:
Respecting user privacy is a core principle of Linux Mint. Unlike some operating systems, Linux Mint does not collect data or telemetry by default, ensuring your personal information remains private.

About Me

Hi, My name is Juma Shafara. Am a Data Scientist and Instructor at DATAIDEA. I have taught hundreds of peope Programming, Data Analysis and Machine Learning. I also enjoy developing innovative algorithms and models that can drive insights and value. I regularly share some content that I find useful throughout my learning/teaching journey to simplify concepts in Machine Learning, Mathematics, Programming, and related topics. Besides these technical stuff, I enjoy watching soccer, movies and reading mystery books.

What's on your mind? Put it in the comments!

Python Quick Review

Juma Shafara — Thu, 25 Jul 2024 09:39:00 +0000

Introduction

Python is a great general-purpose programming language on its own, but with the help of a few popular libraries (numpy, scipy, matplotlib) it becomes a powerful environment for scientific computing.

I expect that you have some experience with Python; and if you don't, this section will serve as a quick crash course

In this tutorial, we will cover:

Basic Python: Basic data types (Containers, Lists, Dictionaries, Sets, Tuples), Functions, Classes

A Brief Note on Python Versions

As of Janurary 1, 2024, Python has officially dropped support for python2. We'll be using Python 3.10 for this iteration of the course. You can check your Python version at the command line by running python --version.

# checking python version
!python --version

Python 3.10.12

Don't Miss Any Updates!

Before we continue, we have a humble request, to be among the first to hear about future updates of the course materials, simply enter your email below, follow us on (formally Twitter), or subscribe to our YouTube channel.

Basics of Python

Python is a high-level, dynamically typed multiparadigm programming language. Python code is often said to be almost like pseudocode, since it allows you to express very powerful ideas in very few lines of code while being very readable. As an example, here is an implementation of the classic quicksort algorithm in Python:

def quicksort(array):
    if len(array) <= 1:
        return array
    pivot = array[len(array) // 2]
    left = [number for number in array if number < pivot]
    middle = [number for number in array if number == pivot]
    right = [number for number in array if number > pivot]
    return quicksort(left) + middle + quicksort(right)

quicksort([3,6,8,10,1,2,1])

[1, 1, 2, 3, 6, 8, 10]

sorted('listen', reverse=True)

['t', 's', 'n', 'l', 'i', 'e']

Variables

Variables are stores of value

name = 'Juma'
age = 19
id_number = 190045

Rules to consider

Variable names should be meaningful eg number instead of x
Variable names should contain only alpha-numberic characters, and maybe under_scores
Variable names can only start with letters or an underscore
Variable name cannot contain special characters
Variables names are case sensitive

Examples of variables

For this course, we'll use snake case for quick variables

name = 'Eva'
list_of_names = ['Eva', 'Shafara', 'Bob'] # snake case

we'll use camel case for function variables

def calculateBMI(weight_kg, height_m): # camel case
    bmi = weight_kg / height_m ** 2
    rounded_bmi = round(bmi, 3)
    return rounded_bmi

finally, we'll use pascal case for class variables

class MathFunction:    # Pascal Case
    def __init__(self, number):
        self.number = number

    def square(self):
        return self.number ** 2

    def cube(self):
        return self.number ** 3

Basic data types

Numbers

Integers and floats work as you would expect from other languages:

number = 3
print('Number: ', number)
print('Type: ', type(number))

Number:  3
Type:  <class 'int'>

# Quick number arithmetics

print(number + 1)   # Addition
print(number - 1)   # Subtraction
print(number * 2)   # Multiplication
print(number ** 2)  # Enumberponentiation

# Some compound assingment operators

number += 1 # number = number + 1
print(number)
number *= 2
print(number)
number /= 1 # number = number / 1
print(number)
number -= 2
print(number)

number = 2.5
print(type(number))
print(number, number + 1, number * 2, number ** 2)

<class 'float'>
2.5 3.5 5.0 6.25

# complex numbers
vector = 2 + 6j
type(vector)

complex

Booleans

Python implements all of the usual operators for Boolean logic, but uses English words rather than symbols:

t, f = True, False
type(t)

bool

Now we let's look at the operations:

# Logical Operators

print(t and f) # Logical AND;
print(t or f)  # Logical OR;
print(not t)   # Logical NOT;
print(t != f)  # Logical XOR;

False
True
False
True

Strings

A string is a sequence of characters under some quotes. Eg.

hello = 'hello'   # String literals can use single quotes
world = "world"   # or double quotes; it does not matter
print(hello, len(hello))

hello 5

# We can string in python
full = hello + ' ' + world  # String concatenation
print(full)

hello world

hw12 = '{} {} {}'.format(hello, world, 12)  # string formatting
print(hw12)

hello world 12

statement = 'I love to code in {}'
modified = statement.format('JavaScript')
print(modified)

I love to code in JavaScript

# formatting by indexing
statement = '{0} loves to code in {2} and {1}'
statement.format('Juma', 'Python', 'JavaScript')

'Juma loves to code in JavaScript and Python'

# formatting by name
statement = '{name} loves to code in {language1} and {language2}'
statement.format(language2='Python', name='Juma', language1='JavaScript')

'Juma loves to code in JavaScript and Python'

# String Literal Interpolation
name = 'Juma'
language1 = 'JavaScript'
language2 = 'Python'

statement = f'{name} loves to code in {language1} and {language2}'

print(statement)

Juma loves to code in JavaScript and Python

String objects have a bunch of useful methods; for example:

string_ = "hello"
print(string_.capitalize())  # Capitalize a string
print(string_.upper())       # Convert a string to uppercase; prints "HELLO"
print(string_.rjust(7))      # Right-justify a string, padding with spaces
print(string_.center(7))     # Center a string, padding with spaces
print(string_.replace('l', '(ell)'))  # Replace all instances of one substring with another
print('  world '.strip())  # Strip leading and trailing whitespace

Hello
HELLO
  hello
 hello 
he(ell)(ell)o
world

statement = 'i love to code in Python '

capitalized = statement.capitalize()
upped = statement.upper()
replaced = statement.replace('Python', 'javascript')
statement.strip()

'i love to code in Python'

You can find a list of all string methods in the documentation.

Containers

Python containers (collections) are objects that we use to group other objects
Python includes several built-in container types: lists, dictionaries, sets, and tuples.

Lists

A list is an ordered collection of python objects or elements. A list can contain objects of different data types

list_of_numbers = [3, 1, 2]   # Create a list
print(list_of_numbers)
print(list_of_numbers[2])
print(list_of_numbers[-1])     # Negative indices count from the end of the list; prints "2"

[3, 1, 2]
2
2

list_of_numbers[2] = 'foo'    # replacing a specific value in a list
print(list_of_numbers)

[3, 1, 'foo']

list_of_numbers.append('bar') # Add a new element to the end of the list
print(list_of_numbers)

[3, 1, 'foo', 'bar']

last_item = list_of_numbers.pop()     # Remove and return the last element of the list
print(last_item)    # returns the last item 
print(list_of_numbers) # Modifies the original list

bar
[3, 1, 'foo']

Research on:

del
remove()

As usual, you can find all the gory details about lists in the documentation.

Slicing

In addition to accessing list elements one at a time, Python provides concise syntax to access a range of values in a list; this is known as slicing:

list_of_numbers = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
print(list_of_numbers)

[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

print(list_of_numbers)         # Prints "[0, 1, 2, 3, 4]"
print(list_of_numbers[2:4])    # Get a slice from index 2 to 4 (exclusive); prints "[2, 3]"
print(list_of_numbers[2:])     # Get a slice from index 2 to the end; prints "[2, 3, 4]"
print(list_of_numbers[:2])     # Get a slice from the start to index 2 (exclusive); prints "[0, 1]"
print(list_of_numbers[:])      # Get a slice of the whole list; prints ["0, 1, 2, 3, 4]"
print(list_of_numbers[:-1])    # Slice indices can be negative; prints ["0, 1, 2, 3]"
list_of_numbers[2:4] = [8, 9] # Assign a new sublist to a slice
print(list_of_numbers)         # Prints "[0, 1, 8, 9, 4]"

[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
[2, 3]
[2, 3, 4, 5, 6, 7, 8, 9]
[0, 1]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
[0, 1, 2, 3, 4, 5, 6, 7, 8]
[0, 1, 8, 9, 4, 5, 6, 7, 8, 9]

Loops

A for loop is used to loop through (or iterate) over a sequence of objects (iterable objects). Iterable objects in python include strings, lists, sets etc

You can loop over the elements of a list like this:

list_of_animals = ['cat', 'dog', 'monkey']

for animal in list_of_animals:
    print(animal)

cat
dog
monkey

list_of_numbers = [1, 2, 3, 4, 5, 6, 7, 8, 9, 0]
list_of_squared_numbers = []

for number in list_of_numbers:
    list_of_squared_numbers.append(pow(number, 2))

list_of_squared_numbers

[1, 4, 9, 16, 25, 36, 49, 64, 81, 0]

If you want access to the index of each element within the body of a loop, use the built-in enumerate function:

animals = ['cat', 'dog', 'monkey']

for index, animal in enumerate(animals):
    print(f'{index}: {animal}')

0: cat
1: dog
2: monkey

List comprehensions:

numbers = [0, 1, 2, 3, 4]
squares = []

for number in numbers:
    squares.append(pow(number, 2))

print(squares)

[0, 1, 4, 9, 16]

You can make this code simpler using a list comprehension:

list_of_numbers = [0, 1, 2, 3, 4]

squares = [pow(number, 2) for number in list_of_numbers]

print(squares)

[0, 1, 4, 9, 16]

List comprehensions can also contain conditions:

numbers = [0, 1, 2, 3, 4]

even_squares = [pow(number, 2) for number in numbers if number % 2 == 0]

print(even_squares)

[0, 4, 16]

Research:

How to combine lists

Dictionaries

A dictionary is an unordered and mutable collection of items
A dictionary is created using curly brackets
Each item in a dictionary contains a key/value pair

# creating a dictionary
person = {
    'first_name': 'Juma',
    'last_name': 'Shafara',
    'age': 51,
    'married': True
}
person

{'first_name': 'Juma', 'last_name': 'Shafara', 'age': 51, 'married': True}

# accessing items in a dictionary
first_name = person['first_name']
last_name = person['last_name']
full_name = first_name + ' ' + last_name

# display
full_name

'Juma Shafara'

# add items to a dictionary
person['hobby'] = 'Coding'
person

{'first_name': 'Juma',
 'last_name': 'Shafara',
 'age': 51,
 'married': True,
 'hobby': 'Coding'}

email = person.get('email', 'email not available')
print(email)

email not available

# modifying a value in a dictionay
person['married'] = False
person

{'first_name': 'Juma',
 'last_name': 'Shafara',
 'age': 51,
 'married': False,
 'hobby': 'Coding'}

# remove an item from a dictionary
person.pop('age')
person

{'first_name': 'Juma',
 'last_name': 'Shafara',
 'married': False,
 'hobby': 'Coding'}

Research:

How to remove an item using the del method
How to iterate over objects in a dictionary
Imitate list comprehension with dictionaries

You can find all you need to know about dictionaries in the documentation.

Sets

A set is an unordered, immutable collection of distinct elements.
A set is created using curly braces
The objects are placed inside the brackets and are separated by commas
As a simple example, consider the following:

animals = {'cat', 'dog'}

print('cat' in animals)   # Check if an element is in a set; prints "True"
print('fish' not in animals)  # prints "True"

True
True

animals.add('fish')      # Add an element to a set

print('fish' in animals) # Returns "True"

print(len(animals))       # Number of elements in a set;

True
3

animals.add('cat')       # Adding an element that is already in the set does nothing
print(len(animals)) 

animals.remove('cat')    # Remove an element from a set
print(len(animals))

3
2

Research:

How to remove with discard()
How to remove with pop()
How to combine sets
How to get the difference between 2 sets
What happens when we have repeated elements in a set

Loops: Iterating over a set has the same syntax as iterating over a list; however since sets are unordered, you cannot make assumptions about the order in which you visit the elements of the set:

animals = {'cat', 'dog', 'fish'}

for index, animal in enumerate(animals):
    print(f'{index}: {animal}')

0: fish
1: cat
2: dog

Set comprehensions: Like lists and dictionaries, we can easily construct sets using set comprehensions:

from math import sqrt

print({int(sqrt(x)) for x in range(30)})

{0, 1, 2, 3, 4, 5}

Tuples

A tuple is an (immutable) ordered list of values.
A tuple is in many ways similar to a list; one of the most important differences is that tuples can be used as keys in dictionaries and as elements of sets, while lists cannot. Here is a trivial example:

d = {(x, x + 1): x for x in range(10)}  # Create a dictionary with tuple keys
t = (5, 6)       # Create a tuple
print(type(t))
print(d[t])       
print(d[(1, 2)])

<class 'tuple'>
5
1

# t[0] = 1

Research:

Creating a tuple
Access items in a tuple
Negative indexing tuples
Using range of indexes
Getting the length of items in a tuple
Looping through a tuple
Checking if an item exists in a tuple
How to combine tuples
Prove that tuples are immutable

Functions

A function is a group of statements that performs a particular task
Python functions are defined using the def keyword. For example:

def overWeightOrUnderweightOrNormal(weight_kg:float, height_m:float) -> str:
    '''
    Tells whether someone is overweight or underweight or normal
    '''
    height_m2 = pow(height_m, 2)
    bmi = weight_kg / height_m2
    rounded_bmi = round(bmi, 3)
    if bmi > 24:
        return 'Overweight'
    elif bmi > 18:
        return 'Normal'
    else:
        return 'Underweight'

overWeightOrUnderweightOrNormal(67, 1.7)

'Normal'

We will often use functions with optional keyword arguments, like this:

bmi = calculateBMI(height_m=1.7, weight_kg=67)

print(bmi)

23.183

def greet(name:str='You')->str:
    """
    This function greets people by name
    Example1:
    >>> greet(name='John Doe')
    >>> 'Hello John Doe'
    Example2:
    >>> greet()
    >>> 'Hello You'
    """
    return f'Hello {name}'

# greet('Eva')
?greet

[0;31mSignature:[0m [0mgreet[0m[0;34m([0m[0mname[0m[0;34m:[0m [0mstr[0m [0;34m=[0m [0;34m'You'[0m[0;34m)[0m [0;34m->[0m [0mstr[0m[0;34m[0m[0;34m[0m[0m
[0;31mDocstring:[0m
This function greets people by name
Example1:
>>> greet(name='John Doe')
>>> 'Hello John Doe'
Example2:
>>> greet()
>>> 'Hello You'
[0;31mFile:[0m      /tmp/ipykernel_66386/2049930273.py
[0;31mType:[0m      function

Classes

In python, everything is an object
We use classes to help us create new object
The syntax for defining classes in Python is straightforward:

class Person:
    first_name = 'John'
    last_name = 'Tong'
    age = 20

# Instantiating a class
object1 = Person()

print(object1.first_name)
print(object1.last_name)
print(object1.age)

print(f'object1 type: {type(object1)}')

John
Tong
20
object1 type: <class '__main__.Person'>

# Instantiating a class
object2 = Person()

print(object2.first_name)
print(object2.last_name)
print(object2.age)

John
Tong
20

class Person:
    def __init__(self, first_name, last_name, age):
        self.first_name = first_name
        self.last_name = last_name
        self.age = age

    def greet(self, name):
        return f'Hello {name}'

object1 = Person('Juma', 'Shafara', 24)
print(object1.first_name)
print(object1.last_name)
print(object1.age)
print(type(object1))

Juma
Shafara
24
<class '__main__.Person'>

object2 = Person('Eva', 'Ssozi', 24)
print(object2.first_name)
print(object2.last_name)
print(object2.age)
print(object2.greet('Shafara'))
print(type(object2))

Eva
Ssozi
24
Hello Shafara
<class '__main__.Person'>

class Student(Person):
    def __init__(self, first_name, last_name, age, id_number, subjects=[]):
        super().__init__(first_name, last_name, age)
        self.id_number = id_number
        self.subjects = subjects

    def addSubject(self, subject):
        self.subjects.append(subject)

student1 = Student('Calvin', 'Masaba', 34, '200045', ['math', 'science'])

student1.addSubject('english')

student1.subjects

['math', 'science', 'english']

Research:

Inheritance: This allows to create classes that inherit the attributes and methods of another class

What's on your mind? Put it in the comments!

Python Classes, Formatted Strings, Errors and Variable Scope

Juma Shafara — Tue, 23 Jul 2024 11:13:12 +0000

In this notebook, we will explore some fundamental concepts in Python that are essential for writing clean, efficient, and maintainable code. These concepts include:

Classes and Objects
Formatted Strings
Handling Errors
Variable Scopes

Don't Miss Any Updates!

Classes and Objects

In Python, everything is an object. A class helps us create objects.

Creating a Class

Use the class keyword to create a class

Here is the syntax:

class className:
    statement(s)

Below is an example:

class Person:
    first_name = "Betty"
    last_name = "Kawala"
    age = 30

Instantiating a class

Now we can ceate an object from the class by instantiating it.

To instantiate a class, add round brackets to the class name.

person_obj1 = Person()

type(person_obj1)

__main__.Person

After instantiating a class, we can now access the object's properties.

# print attributes
print(person_obj1.first_name)
print(person_obj1.last_name)
print(person_obj1.age)

Betty
Kawala
30

Class Attributes

A class can have attributes. Forexample the Person Class can have attributes like the name, height and feet

class Person:
    def __init__(self, name, height, feet):
        self.name = name
        self.height = height
        self.feet = feet

Note!

For now, focus on the syntax. Later we will explain the init() function and the self parameter.

Now that our class is ready, we can now instantiate it and provide values to it's attributes.

This process can also be called "creating an instance of a class".

An instance is simply the object created from a class

In this example, person_obj1 is a unique instance of the person class.

# create a class instance
person_obj = Person(
    name='Betty Kawala', 
    height=1.57, 
    feet=4
    )

After that, we can now access the properties of the instance (object)

# accessing the properties
print('Name:', person_obj.name)
print('Height:', person_obj.height)
print('Feet:', person_obj.feet)

Name: Betty Kawala
Height: 1.57
Feet: 4

The self parameter allows us to access the attributes and methods of a class

The __init__() function allows us to provide values for the attributes of a class

Instances are unique

Let's say you have 500 people and you need to manage their data.

It is inefficient to create a variable for each of them, instead, you can create unique instances of a class.

In this example, the student1 and student2 instances are different from each other

class Student:
  def __init__(self, id_number, name, age):
    self.id_number = id_number
    self.name = name
    self.age = age

student1 = Student(5243, "Mary Doe", 18)
student2 = Student(3221, "John Doe", 18)

print("Student 1 ID:", student1.id_number)
print("Student 1 Name:", student1.name)
print("Student 1 Age:", student1.age)

print("---------------------")

print("Student 2 ID:", student2.id_number)
print("Student 2 Name:", student2.name)
print("Student 2 Age:", student2.age)

Student 1 ID: 5243
Student 1 Name: Mary Doe
Student 1 Age: 18
---------------------
Student 2 ID: 3221
Student 2 Name: John Doe
Student 2 Age: 18

Methods

Methods are functions that can access the class attributes.

These methods should be defined (created) inside the class

class Person:
    def __init__(self, name, height, feet):
        self.name = name
        self.height = height
        self.feet = feet

    def jump(self):
        return "I'm jumping " + str(self.feet) + " Feet"

person_obj1 = Person(name='Juma', height=1.59, feet=5)

print(person_obj1.jump())

I'm jumping 5 Feet

As you may notice, we used the self parameter to access the feet attribute

You can also pass an argument to a method.

class Student:
    def __init__(self, id_number, name, age):
        self.id_number = id_number
        self.name = name
        self.age = age

    def greet_student(self, greetings):
        print("Hello" + self.name + ", " + greetings)

student1 = Student(43221, "Agaba Calvin", 18)

# the string below will be passed as 
# the value of the greetings parameter
student1.greet_student("Welcome to this Python Tutorial!")

HelloAgaba Calvin, Welcome to this Python Tutorial!

Python Inheritance

Inheritance is a feature that allows us to create a class that inherits the attributes or properties and methods of another class

Example

The Animal class below can be used to tell that an animal can eat

class Animal:
    def __init__(self, name, age):
        self.name = name
        self.age = age

    def eat(self):
        print(f"{self.name} is eating.")

Let's say we need to create another class called Dog.

Since a dog is also an animal, it's more efficient to have access to all the properties and methods of the Animal class than to create another

This example creates a class named Dog and inherits from the Animal class

class Dog(Animal):
    def __init__(self, name, age, color):
        super().__init__(name, age)
        self.color = color

    def sound(self):
        print(self.name, "barks")

Note!

As you may notice, to inherit from a parent, we simply pass the name of that class as a parameter of the child class.

Now we can use the properties and methods of both the Animal and the Dog classes using just one instance

dog1 = Dog(name='Brian', age=8, color='White')
dog1.eat()
dog1.sound()

Brian is eating.
Brian barks

The super() and __init__ functions found in the Dog class allow us to inherit the properties and methods of the Animal class.

Parent and Child Class

The parent class is the class from whick the other class inherits from.

The child class is the the class that inherits from another class

In our example above, the Animal is the parent class while the Dog class is the child class

Formatted Strings

In Python, we can format a string by adding substring/s within it.

The format() function allows us to format strings.

Placeholders `{}`

Placeholders help us control which part of the string should be formated.

They are defined using curly braces {}.

In this example, we will concatenate (add) a substring to where the curly braces are placed

text = "I love {} very much!"
formatted_text = text.format("Python")

print(formatted_text)

I love Python very much!

Multiple placeholders

If you want to format multiple parts of a string, use multiple placeholders.

text = '{} loves to code in {}'
formatted_text = text.format('Juma', 'JavaScript')

print(formatted_text)

Juma loves to code in JavaScript

Using Indexes

We can use index numbers to specify exactly where the values should be placed.

The index numbers should be inside the curly braces: {index_numbers}

text = 'I love {2}, {1} and {0} very much!'
formatted_text = text.format('Python', 'JavaScript', 'HTML')

print(formatted_text)

I love HTML, JavaScript and Python very much!

Note!

0 represents the first value, 1 represents the second value and so on.

Using Named Indexes

We can also use named indexes to specify exactly where the values should be placed.

The arguments of the format() function should be in key/value pairs ie key=value.

The key/value pairs should be separated by commas.

text = 'The color of the {fruit} is {color}.'
formatted_text = text.format(fruit='banana', color='yellow')

print(formatted_text)

The color of the banana is yellow.

Literal String Interpolation

Literal string interpolation allows you to use expression inside your strings.

Simply add f before you opening quote, then surround your expressions with curly braces {}.

name = 'Juma'; 
language = 'JavaScript'

statement = f'{name} loves to code in {language}'

print(statement)

Juma loves to code in JavaScript

Here's another example

number1 = 5
number2 = 7
answer = f'The summation of 5 and 7 is {number1 + number2}'

print(answer)

The summation of 5 and 7 is 12

Errors in Python

When coding in Python, you will encounter errors.

When errors occur, the program crashes or stops executing.

Fortunately, errors can be handled in Python

The `try...except` statment

The try...except statement is used to handle exceptions(errors)

The try statement takes a block of code to test for errors

The except statement handles the exceptions.

try:
    # age = input('Enter your age: ')
    age = '32'

    if age >= 18:
        print('Your vote has been cast')
    else:
        print('You are not eligible to vote')
except:
    print('A problem occured while picking your age \n'
          'You did not enter a number')

text = "\nHello world!"
print(text)

A problem occured while picking your age 
You did not enter a number

Hello world!

Note!

Even when the exception was thrown, the codes after the try...except were still executed

The `else` statement

The else statement is executed if there are no exceptions thrown.

try:
    text = "Hello World!"
    print(text)
except:
    print("An error occurred.")
else:
    print("No exception was thrown!")

Hello World!
No exception was thrown!

The `finally` statement

The finally statement is executed whether or not an exception is thrown.

try: 
    text = "hello world"
    print(text)
except:
    print("An error occured.")
finally:
    print("Hello, I am still printed.")

print("----------------------")

# an exception will be thrown here
try: 
    print(undefined_variable)
except:
    print("An error occured.")
finally:
    print("Hello, I am still printed.")

hello world
Hello, I am still printed.
----------------------
An error occured.
Hello, I am still printed.

Throw Exceptions

We can intentionally throw and exception to stop the execution of a program.

The raise keyword throws an excrption.

# Creating your own errors
try: 
    age = 14
    if age < 18:
        raise Exception('Not an adult')
except Exception as error:
    print('A problem occurred \n'
          f'Error: {error}')

A problem occurred 
Error: Not an adult

Kinds of Exceptions

In Python, there are different kinds of exceptions and we can handle them individually with the try...except statement.

try:
    # statements
except ExceptionKind:
    #statments

One of the most common kind of exceptions is the NameError. This is thrown when you use a variable that is not defined

try:
    print(rand_var)
except NameError:
    print('You used a variable that is not defined!')

You used a variable that is not defined!

Variable Scope

Python Variable Scopes

The accessibility of variable depends on its scope. In Python, there are two variable scopes:

Global Scope
Local Scope

Global Scope

A variable that is defined (created) outside a function has a global scope

A global variable can be accessed anywhere in a program

name = 'Viola'
# name can be accessed here
print(name)

def greet():
    # name can be accessed here
    print('Hello ' + name)

greet()

Viola
Hello Viola

Local Scope

A variable that is defined (created) inside a function has a local scope. A local scope variable can only be accessed and used inside the function.

def greet():
    local_name = 'Viola'
    print('Hello ' + local_name)

greet()

try:
    # local_name cannot be accessed here
    print(local_name)
except Exception as e:
    print(e)

Hello Viola
name 'local_name' is not defined

The `global` Keyword

We can force a local variable to be a global variable by using the global keyword.

# Global Keyword

def add():
    global summ
    number1 = 5
    number2 = 7
    summ = number1 + number2
    return summ

add()

# summ is accessible even outside the function
print(summ)

What's on your mind? Put it in the comments!

Python Flow Control Full Tutorial

Juma Shafara — Wed, 17 Jul 2024 14:03:56 +0000

Python control flow tools change the flow of how code is executed by the Python interpreter.

Since the Python interpreter executes code in a line-by-line manner, Python control flow tools help dictate what line(s) of code should run in a Python program. There are different types of control flow tools available to us in Python and we will go through them in detail in this lesson.

Functions

A function in python is a group statements that perform a particular task

# This function calculates Body Mass Index
def calculateBodyMassIndex(weight_kg, height_m):

    body_mass_index = weight_kg / pow(height_m, 2)
    rounded_bmi = round(body_mass_index, 2)

    return rounded_bmi

# lets try
calculateBodyMassIndex(67, 1.6)

26.17

Creating a function

To create a function, we need the following:

The def keyword
A function name
Round brackets () and a colon :
A function body- a group of statements

def greeter():
    message = 'Hello'
    print(message)

To execute a function, it needs to be called
To call a function, use its function name with parentheses ()

greeter()

Hello

Function Parameters/Arguments

When calling a function, we can pass data using parameters/arguments
A parameter is a variable declared in the function. In the example below, number1 and number2 are parameter
The argument is the value passed to the function when its called. In the example below 3 and 27 are the arguments

# define the function
def addNumbers(number1, number2):
    sum = number1 + number2
    print(sum)

# Call the function
addNumbers(3, 27)

Default Arguments:

A function can have default arguments.

It can be done using the assignment operator (=).

If you don't pass the argument, the default argument will be used instead.

def hello(name = 'Agaba'):
    print('Hello ' + name)

hello('John') # calling with John
hello() # calling with no name

Hello John
Hello Agaba

Return Statement

The return statement is used to return a value to a function caller

def addNumbers(number1, number2):
    sum = number1 + number2
    return sum

summation = addNumbers(56, 4)
print(summation)

Important!

The return statement stops the execution of a function.

### lambda functions

Lambda functions (also called anonymous functions) are functions that donot have names
The body of a lambda function can only have one expression, but can have multiple arguments
The result of the expression is automatically returned

Syntax:

  lambda parameters: expression

# Example of lambda function
calculateBMI = lambda weight_kg, height_m: round((weight_kg/(height_m ** 2)), 2)

# Calling a labda function
calculateBMI(67, 1.7)

23.18

Note!

In the example above, the body mass index is automatically return, even without using the return statement

Practice functions

Calculate CGPA

# Assume 4 course units
# 1. Math - A
# 2. Science - B
# 3. SST - B
# 4. English - C


def calculate_CGPA(GPs_list, CUs_list):
    length = len(GPs_list)
    product_sum = 0

    for item in range(length):
        product_sum += GPs_list[item] * CUs_list[item]

    CUs_sum = sum(CUs_list)

    CGPA = product_sum / CUs_sum

    return CGPA

# calculate_CGPA(4, 5)

Get someones age given birth month and year

def getAge(month, year):
    month_diff = 12 - month
    year_diff = 2023 - year

    return str(year_diff) + ' years ' + str(month_diff) + ' months'  

age = getAge(year=2000, month=10) # keyword argument
age2 = getAge(10, 2000) # positional argument

print(age)

23 years 2 months

Don't Miss Any Updates!

Loops

Loops are used to repetitively execute a group of statements
we have 2 types, for and while loop

For Loop

A for loop is used to loop through or iterate over a sequence or iterable objects

Syntax:

for variable in sequence:
    statements

Looping through a list

The for loop is commonly used with lists.

pets = ['cat', 'dog', 'rabbit']
# iterate through pets
for pet in pets:
    print(pet)

cat
dog
rabbit

Here's another example to convert many weights from kilograms(kgs) to pounds(pds)

# convert all weights in list from kg to pounds
weights_kg = [145, 100, 76, 80]
weights_pds = []

for weight in weights_kg:
    pounds = weight * 2.2
    rounded_pds = round(pounds, 2)
    weights_pds.append(rounded_pds)

print(weights_pds)

This example displays all letters in my name.

# Display all letters in a name
name = 'Shafara'

for letter in name:
    print(letter)

S
h
a
f
a
r
a

While loop

The while loop executes a given group of statements as long as the given expression is True

Syntax:

while expression:
    statements

counter = 0

while counter < 5:
    print('Hello you')
    counter += 1

Hello you
Hello you
Hello you
Hello you
Hello you

# Convert the weights in the list from kgs to pounds
weights_kg = [145, 100, 76, 80]
weights_pds = []

counter = 0
end = len(weights_kg)

while counter < end:

    pound = weights_kg[counter] * 2.2
    rounded_pds = round(pound, 3)
    weights_pds.append(rounded_pds)

    counter += 1

print(weights_pds)

[319.0, 220.0, 167.2, 176.0]

Conditional Statements

Conditional statements in Python are fundamental building blocks for controlling the flow of a program based on certain conditions. They enable the execution of specific blocks of code when certain conditions are met. The primary conditional statements in Python include if, elif, and else.

Basic Syntax

If Statement

The if statement is used to test a condition. If the condition evaluates to True, the block of code inside the if statement is executed.

if condition:
    # block of code

Example:

x = 10
if x > 5:
    print("x is greater than 5")

Else Statement

The else statement is used to execute a block of code if the condition in the if statement evaluates to False.

if condition:
    # block of code if condition is True
else:
    # block of code if condition is False

Example:

x = 3
if x > 5:
    print("x is greater than 5")
else:
    print("x is not greater than 5")

Elif Statement

The elif (short for else if) statement allows you to check multiple conditions. If the first condition is False, it checks the next elif condition, and so on. If all conditions are False, the else block is executed.

if condition1:
    # block of code if condition1 is True
elif condition2:
    # block of code if condition2 is True
else:
    # block of code if none of the above conditions are True

Example:

x = 7
if x > 10:
    print("x is greater than 10")
elif x > 5:
    print("x is greater than 5 but less than or equal to 10")
else:
    print("x is 5 or less")

Nested Conditional Statements

Conditional statements can be nested within each other to handle more complex decision-making processes.

Example:

x = 15
if x > 10:
    if x > 20:
        print("x is greater than 20")
    else:
        print("x is greater than 10 but not greater than 20")
else:
    print("x is 10 or less")

Conditional Expressions (Ternary Operator)

Python also supports conditional expressions, which allow for a more concise way to write simple if-else statements.

variable = value_if_true if condition else value_if_false

Example:

x = 10
result = "greater than 5" if x > 5 else "5 or less"
print(result)  # Output: greater than 5

Combining Conditions

Multiple conditions can be combined using logical operators (and, or, not).

Example:

x = 8
if x > 5 and x < 10:
    print("x is between 5 and 10")

Practical Usage

Conditional statements are used in a wide variety of scenarios, such as:

Validating user input.
Controlling the flow of loops.
Implementing different behaviors in functions or methods.
Handling exceptions or special cases in data processing.

Understanding and effectively using conditional statements are crucial for writing efficient and readable code in Python. They enable developers to build programs that can make decisions and respond dynamically to different inputs and situations.

What's on your mind? Put it in the comments!

Creating Image Frames from Videos for Deep Learning Models

Juma Shafara — Tue, 07 Feb 2023 08:57:55 +0000

Introduction

In recent years, deep learning has been widely adopted in various fields, including computer vision, natural language processing, and robotics. The success of deep learning models in these fields is largely due to their ability to learn and recognize patterns and features in large datasets. To train these models, large amounts of high-quality data are required. In the context of computer vision, image data is a crucial ingredient in the training process.

One of the challenges in computer vision is to process and analyze video data. Videos consist of a sequence of images, and extracting relevant information from a video can be challenging. However, extracting frames from videos and using them as inputs for deep learning models is an effective solution. By creating image frames from videos, we can train deep-learning models to detect and classify objects and events in real time.

This article provides an overview of why and how to create image frames from videos for deep learning models. We will outline the steps involved in extracting frames from videos and preparing them as input for a deep-learning model.

Why create image frames from videos?

Videos contain a wealth of information that can be useful for deep learning models. However, videos can be challenging to process, as they are large in size, contain a large number of frames, and may be of varying quality. By creating image frames from videos, we can overcome these challenges and train deep learning models more effectively.

Image frames allow us to focus on specific moments in the video and eliminate irrelevant information. They also make it easier to pre-process and augment the data, which can improve the performance of the deep learning model. Additionally, image frames are easier to store and manage than videos, as they take up less space and are easier to manipulate.

How to create image frames from a video using OpenCV?

OpenCV (Open Source Computer Vision Library) is an open-source computer vision and machine learning software library. It is widely used for computer vision tasks, such as image and video processing, object detection and recognition, and more.

Below is the code to generate image frames from videos using OpenCV

import cv2

# Load the video
video = cv2.VideoCapture("path/to/video.mp4")

# Get the frame count
frame_count = int(video.get(cv2.CAP_PROP_FRAME_COUNT))

# Extract the frames at a specified frequency
frequency = 30  # Extract a frame every 30 frames
for i in range(frame_count):
    # Read a frame from the video
    ret, frame = video.read()

    # Check if the frame was successfully read
    if not ret:
        break

    # Save the frame if it is a specified frequency
    if i % frequency == 0:
        filename = "path/to/output/folder/frame_{}.jpg".format(i)
        cv2.imwrite(filename, frame)

# Release the video
video.release()

How to preprocess the image frames for a deep-learning model?

Preprocessing image frames for a deep learning model is crucial for achieving good performance. The following are the steps for preprocessing image frames for a deep-learning model:

Resizing the images to the same size: Deep learning models typically require that the input images have the same size, so the images should be resized to the same size. This can be done using OpenCV's resize function.
Normalizing the pixel values: Deep learning models work best when the pixel values are normalized, so the pixel values should be normalized to the range [0, 1]. This can be done by dividing each pixel value by 255.
Converting the images to a tensor: Deep learning models work with tensors, so the images should be converted to tensors. This can be done using the to_tensor function from the PyTorch library or convert_to_tensor from the Tensorflow library.
Splitting the data into training and validation sets: To evaluate the performance of the model, the data should be split into training and validation sets. This can be done using the train_test_split function from the sci-kit-learn library.

The following is a sample Python code that implements the above preprocessing steps:

import cv2
import numpy as np
import torch
from sklearn.model_selection import train_test_split

def preprocess_image(image):
    # resize the image
    image = cv2.resize(image, (224, 224))
    # normalize the pixel values
    image = image / 255.0
    # convert the image to a tensor
    image = torch.from_numpy(image).float()
    return image

# load the images from the folder
images = []
for i in range(n):
    # read the i-th image
    image = cv2.imread(f'path/to/output/folder/frame_{i}.jpg')
    # preprocess the image
    image = preprocess_image(image)
    images.append(image)

# convert the list of images to a tensor
images = torch.stack(images)

# split the data into training and validation sets
train_images, val_images = train_test_split(images, test_size=0.2, random_state=42)

By following these steps, the images are ready to be used as input for a deep-learning model.

Conclusion
Extracting image frames from videos is crucial in training deep-learning models for object detection and classification tasks. The OpenCV library provides a simple and efficient way to extract frames from videos and prepare them for training a deep learning model. By generating image frames, we can have a larger dataset, fine-tune our model, and evaluate its performance, leading to improved results.

How to prepare your custom images dataset for a deep learning model

Juma Shafara — Wed, 23 Nov 2022 16:57:20 +0000

Almost all the deep-learning tutorials out there teach how to build models using already set or inbuilt datasets, but what if you want to train a model to recognize you and maybe your family's faces, a dataset for that purpose is not readily available on the internet.
The ability to organize and build your own image dataset to solve your problem is a demonstration of expertise.
In this article, you will learn how to prepare your own dataset of raw images, which you can then use for your own image classification/computer vision projects.

Steps

Gather and organize your images.

For example, if I want to build a model that recognizes humans(faces) and nonhumans, I put all the human images in one folder and all the nonhuman images in another folder. The two folders are then put into one main folder.

Open a jupyter notebook and import the cv2, PIL, and pathlib.

import cv2
import PIL
import pathlib
import numpy as np

You can find out more about cv2 here, or about PIL here and about pathlib here

Specify the route to the folder and create a Path object from it

images_dir = "/path/to/your/images_directory"

images_dir = pathlib.Path(images_dir)

Create two objects, one containing the image classes lists and another containing the class labels

class_images = {
    'human': list(images_dir.glob('human/*')),
    "nonhuman": list(images_dir.glob("nonhuman/*"))
}

class_labels = {
    "human": 0,
    "nonhuman": 1
}

Initialize X and y to empty python lists. Convert each of the images to a numpy array, resize it to a standard size and append it to the X list. Also, append the corresponding image class label to the y list.

X, y = [], []

for image_name, images in class_images.items():
  print(class_images.items())
  for image in images:
    img = cv2.imread(str(image))
    resized_img = cv2.resize(img, (256, 256))
    X.append(resized_img)
    y.append(class_labels[image_name])

Finally convert X and y to numpy arrays and split each into train and test sets

X = np.array(X)
y = np.array(y)

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)

There you go, now you have your custom image train and test datasets, however, if you want your model to perform better and learn quicker, there are a few other things you may want to do for your image datasets, for example, rescaling and augmentation. These are however out of the scope of this article, if you are interested in that subscribe to my newsletter so that you don't miss out on the upcoming article that will cover how to do them.

Also, follow me on twitter(juma_shafara) where I share other programming stuff.

Deploy Machine Learning models with Django(the easy way)

Juma Shafara — Sat, 22 Oct 2022 01:08:46 +0000

The goal of building a machine learning application is to solve a problem, and an ML model can only do this when it is actively being used in production. As such, ML model deployment is just as important as ML model development.

Deployment is the process by which an ML model is moved from an offline environment and integrated into an existing production environment, such as a live application. It is a critical step that must be completed in order for a model to serve its intended purpose and solve the challenges it is designed for.

A common complaint among ML teams, however, is that deploying ML models in production is a complicated process. It is such a widespread issue that some experts estimate that as many as 90 percent of ML models never make it into production in the first place.

In this article, I will show you the basic way to deploy a machine learning model using a really popular python web framework, Django. We will build a model that recommends a music genre to someone depending on their age and gender. I am assuming you already know about ML model development and are looking for a way to deploy them.

The dataset.

Below is the dataset we will use to train our model, to easily follow along, save it as music.csv

Step 1: Fit the model and store it in a joblib file.

If you are quite familiar with machine learning, the code below is one of the simplest cases so should not be difficult to understand.

import joblib
import pandas as pd
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split

data = pd.read_csv("music.csv")

X,y = data.drop(columns= ["genre"]), data.genre

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size= 0.3)

dtc = DecisionTreeClassifier()

dtc.fit(X_train, y_train)

joblib.dump(dtc, "premus.joblib")

What is happening?

First and foremost, we install packages pandas, which we use later to read our .csv dataset, DecisionTreeClassifier which is the algorithm we will use to fit the model, train_test_split will be used to split the data into training and testing sets.

Secondly, we load the data, from which we select the predictor variables and outcome variables, X and y respectively.

Next up, we then split the data into training and test sets to avoid over-fitting. We then initialize the model and train it (fitting with the training set).

Finally, store a binary file of our model that we will use later to make predictions, we use joblib to do this (dump).

Step 2: Set up Django

Django is a high-level Python framework for creating scalable and robust web applications. In python, this is the most widely used framework.

The most efficient method to integrate machine learning models in Django is to create a communication interface between the ML model and the web interface. It will acquire data for the model, which will then process it on its own. This interface will navigate you back to the web application's end once you've received the prediction from the model. We can do this through REST APIs or WebSocket.

Install Django

python -m pip install Django

Create the project

django-admin startproject premus

This will create a "premus" directory in your current directory. If it didn’t work, see Problems running django-admin. premus is just a name I chose for our app to mean music-prediction

The new startproject created should appear something like this

premus/
    manage.py
    premus/
        __init__.py
        settings.py
        urls.py
        asgi.py
        wsgi.py

Add templates

We then work on templates, the HTML code through which the user will enter data and the prediction will be shown. So, we will require base.html page.

The base.html will contain an HTML form that will take all the necessary fields as input and upon submitting the form, the user will be redirected back with a recommendation displayed.

The new project structure should look something like this (the order of the files and folders may change)

premus/
    manage.py
    premus/
        __init__.py
        settings.py
        urls.py
        asgi.py
        wsgi.py
    templates/
        form.html

Tell Django where to read the templates from

To do this, add "templates" to the DIRS in the object inside the TEMPLATES list in settings.py

TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': ["templates"],  # right here
        'APP_DIRS': True,
        'OPTIONS': {
            'context_processors': [
                'django.template.context_processors.debug',
                'django.template.context_processors.request',
                'django.contrib.auth.context_processors.auth',
                'django.contrib.messages.context_processors.messages',
            ],
        },
    },
]

Add code to base.html

Add the code below to base.html to collect the fields required. We will not be styling our HTML templates, for now, I just want us to create something functional, you can always do the styling later.

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>Music Recommender</title>
  </head>
  <body>
    <header>
      <nav></nav>
    </header>
    <section>
      <h1>Select to options to get Recommendation</h1>
      <form action="{% url 'prediction' %}" method="post">
        {% csrf_token %}
        <div>
          <label>
            Male
            <input type="radio" name="gender" value="1" />
          </label>
          <label>
            Female
            <input type="radio" name="gender" value="0" />
          </label>
        </div>
        <div>
          <label>
            Age
            <input type="number" name="age" placeholder="eg. 19" />
          </label>
        </div>
        <input type="submit" value="submit and get recommendation" />
      </form>
      <div class="output">
        <h2>Prediction= {{prediction}}</h2>
      </div>
      <div class="output">
        <h3>Age= {{settings.age}}</h3>
        <h3>Gender= {{settings.gender}}</h3>
    </section>
  </body>
</html>

Integrating the music recommender model in a Django project

Copy your trained model dump (the joblib file) and paste it into the inner premus directory, at the end, the setup should look something like this (the order of files may be different due to sorting by alphabetical order).

premus/
    manage.py
    premus/
        __init__.py
        settings.py
        urls.py
        asgi.py
        wsgi.py
        premus.joblib
    templates/
        form.html

Create views.py and urls.py

Now we need to write some code in urls.py, this code lets the computer know which link to show what page

from django.contrib import admin
from django.urls import path
from . import views

app_name = "premus"
urlpatterns = [
    path('admin/', admin.site.urls),
    path('', views.index, name="index"),
    path('/prediction', views.prediction, name="prediction")
]

The views.py will contain code on how to process the user-entered information.

from django.shortcuts import render
import joblib
from pathlib import Path


def index(request):
    return render(request, "base.html", {"prediction": "None"})


def prediction(request):
    BASE_DIR = Path(__file__).resolve().parent.parent
    premus = joblib.load(BASE_DIR / "premus/premus.joblib")
    gender, age = request.POST["gender"], request.POST["age"]
    prediction = premus.predict([[age, gender]])
    settings = {"gender": gender, "age": age}
    return render(request, "base.html", {"prediction": prediction[0], "settings": settings})

Step 3: Start the server and test out

To start the Django development server, navigate to the root of your Django project(where there is manage.py) and run the following command using the terminal

python manage.py runserver

If you enter the URL provided (most likely http://127.0.0.1:8000/) into the browser, the form the user will have to enter data and the results will look something like the below screenshot.

You can extend this to do some more efficient stuff on your other exciting ML project. If you learned something, follow and subscribe to my newsletter for more content like this. Follow me on Twitter

The easiest way to send emails from the server

Juma Shafara — Thu, 15 Sep 2022 05:49:06 +0000

Web sites and applications allow for communication between service providers and their clients. You have probably noticed almost every website has a contact form. In this tutorial, am showing you the simplest way to send emails over a NodeJS backend so you can soon implement it for your clients or on your personal website using the Nodemailer module.

1. What do we need?

You need to have NodeJS installed, if you don't, you can download it from this link

2. What is Nodemailer?

From the official website

Nodemailer is a module for Node.js applications to allow easy-as-cake email sending.
The project got started back in 2010 when there was no sane option to send email messages, today it is the solution most Node.js users turn to by default

3. How to set up Nodemailer?

To install nodemailer, we use npm(node package manager). npm is automatically installed with NodeJS.
Run the code below in your command prompt

npm install nodemailer

When it's done downloading, it can now be included in your application as below

const nodemailer = require("nodemailer");

3. How to send an email?

First, you need to set up a transporter, this is going to be an object that is able to send mails
Secondly, you need to set up your mailData, you can call it anything else, it is an object that defines the mail content, read here for more information
Lastly, we need to call the tell the transporter to send our mailData. Easy

4. Easy example

The example below shows how to use your Gmail account to send an email.

const nodemailer = require('nodemailer');

const transporter = nodemailer.createTransport({
  service: 'gmail',
  auth: {
    user: 'youremail@gmail.com',
    pass: 'your_app_password'
  }
});

const mailData = {
  from: 'youremail@gmail.com',
  to: 'myfriend@yahoo.com',
  subject: 'Sending Email using Node.js',
  text: 'That was easy!'
};

transporter.sendMail(mailData, (error, info) => {
  if (error) {
    console.log(error);
  } else {
    console.log('Email sent: ' + info.response);
  }
});

The pass app password is generated from Gmail, if you don't know how to generate the app password, watch this 2-minute video to guide you.

5. How to send to many recipients at once?

Many times there is a need to send the email to many people, maybe the Boss and other team members.
To send an email to more than one receiver, add them to the "to" property of the mailData object, separated by commas as shown below:

Simple example: Sending to multiple recipients.

const mailData = {
  from: 'youremail@gmail.com',
  to: 'myfriend@yahoo.com, myotherfriend@yahoo.com',
  subject: 'Sending Email using Node.js',
  text: 'That was easy!'
}

6. How to send HTML?

I used to receive well-designed HTML pages from those big organizations, if you've received them too and been wondering how they do it, here you go.
To send HTML formatted text in your email, use the "html" property instead of the "text" property as shown below:

Simple example: Sending HTML

var mailData = {
  from: 'youremail@gmail.com',
  to: 'myfriend@yahoo.com',
  subject: 'Sending Email using Node.js',
  html: '<h1>Welcome</h1><p>That was easy!</p>'
}

If you found this little content use, consider subscribing so that you do not miss out on my upcoming articles, thank you so much.

React (Native) Navigation

Juma Shafara — Wed, 27 Jul 2022 03:28:00 +0000

Introduction

Programming can be disturbing sometimes as you are required to remember a lot of concepts in one project especially if you are just beginning.

An approach I like to use to save myself stress is to just know and understand what am supposed to do, then I care about how to do it later. As you can guess google and cheat sheets have been fundamental to my development.

When I was learning react navigation from the documentation, I thought it wise to save the summaries of each important section for my personal reference so I do not have to run to the documentation every time I had forgotten something. Now I have decided to share with anyone out there that may need such a summary

This is not a how to and you are expected to know what you are doing but just looking to remind yourself, it is just a summary of the basic components and settings you need setting up react navigation in your app. If you are totally blank on react navigation, follow read the react navigation docs instead.

Hello React Navigation

React Native doesn't have a built-in API for navigation like a web browser does. React Navigation provides this for you, along with the iOS and Android gestures and animations to transition between screens.
Stack.Navigator is a component that takes route configuration as its children with additional props for configuration and renders our content.
Each Stack.Screen component takes a name prop which refers to the name of the route and component prop which specifies the component to render for the route. These are the 2 required props.
To specify what the initial route in a stack is, provide an initialRouteName as the prop for the navigator.
To specify screen-specific options, we can pass an options prop to Stack.Screen, and for common options, we can pass `screenOptions to Stack.Navigator

Moving Between Screens

navigation.navigate('RouteName') pushes a new route to the native stack navigator if it's not already in the stack, otherwise it jumps to that screen.
We can call navigation.push('RouteName') as many times as we like and it will continue pushing routes.
The header bar will automatically show a back button, but you can programmatically go back by calling navigation.goBack(). On Android, the hardware back button just works as expected.
You can go back to an existing screen in the stack with navigation.navigate('RouteName'), and you can go back to the first screen in the stack with navigation.popToTop().
The navigation prop is available to all screen components (components defined as screens in route configuration and rendered by React Navigation as a route).

Passing Parameters to Routes

navigate and push accept an optional second argument to let you pass parameters to the route you are navigating to. For example: navigation.navigate('RouteName', { paramName: 'value' }).
You can read the params through route.params inside a screen
You can update the screen's params with navigation.setParams
Initial params can be passed via the initialParams prop on Screen
Params should contain the minimal data required to show a screen, nothing more

Customizing the header bar

You can customize the header inside of the options prop of your screen components. Read the full list of options in the API reference.
The options prop can be an object or a function. When it is a function, it is provided with an object with the navigation and route prop. You can also specify shared screenOptions in the stack navigator configuration when you initialize it. The prop takes precedence over that configuration

Header buttons

You can set buttons in the header through the headerLeft and headerRight properties in options.
The back button is fully customizable with headerLeft, but if you just want to change the title or image, there are other options for that — headerBackTitle, headerBackTitleStyle, and headerBackImageSource.
You can use a callback for the options prop to access navigation and route objects.

Hopefully, you found this useful, am still learning react native and I'll be sharing more or better content about it and other stuff, so make sure to subscribe to my newsletter so that you don't miss out on my other articles.

DEV Community: Juma Shafara

DATAIDEA WALLPAPER

Descriptive Statistics

Descriptive Statistics and Summary Metrics

Don't Miss Any Updates!

What is Descriptive Statistics

1. Measures of central tendency ie Mean, Median, Mode:

2. Measures of variability

Summary

Note!

Note!

Summary

Frequency Tables

Further Reading

What's on your mind? Put it in the comments!

Why Learning Programming Can Be Hard and How to Make It Easier

Why Is Learning Programming So Hard?

How to Make Learning Programming Easier

Final Thoughts

Linux Mint, A Comprehensive Overview

A User-Friendly Interface

Out-of-the-Box Experience

Stability and Performance

Ease of Use and Configuration

Community and Support

Customization and Flexibility

Security and Privacy

About Me

What's on your mind? Put it in the comments!

Python Quick Review

Introduction

A Brief Note on Python Versions

Don't Miss Any Updates!

Basics of Python

Variables

Rules to consider

Examples of variables

Basic data types

Numbers

Booleans

Strings

Containers

Lists

Slicing

Loops

List comprehensions:

Dictionaries

Sets

Tuples

Functions

Classes

What's on your mind? Put it in the comments!

Python Classes, Formatted Strings, Errors and Variable Scope

Don't Miss Any Updates!

Classes and Objects

Creating a Class

Instantiating a class

Class Attributes

Note!

Instances are unique

Methods

Python Inheritance

Example

Note!

Parent and Child Class

Formatted Strings

Placeholders {}

Multiple placeholders

Using Indexes

Note!

Using Named Indexes

Literal String Interpolation

Errors in Python

The try...except statment

Note!

The else statement

The finally statement

Throw Exceptions

Kinds of Exceptions

Variable Scope

Placeholders `{}`

The `try...except` statment

The `else` statement

The `finally` statement

The `global` Keyword