DEV Community: Dana

Maximize Your Python Code: Efficient Serialization and Parallelism with Joblib

Dana — Mon, 24 Jun 2024 12:54:14 +0000

Joblib is a Python library designed to facilitate efficient computation and useful for tasks involving large data and intensive computation.

Joblib tools :

Serialization: Efficiently saving and loading Python objects to and from disk. This includes support for numpy arrays, scipy sparse matrices, and custom objects.
Parallel Computing: Parallelizing tasks to utilize multiple CPU cores, which can significantly speed up computations.

Using Python for Parallel Computing

Threading: The threading module allows for the creation of threads. However, due to the GIL, threading is not ideal for CPU-bound tasks but can be useful for I/O-bound tasks.
Multiprocessing: The multiprocessing module bypasses the GIL by using separate memory space for each process. It is suitable for CPU-bound tasks.
Asynchronous Programming: The asyncio module and async libraries enable concurrent code execution using an event loop, which is ideal for I/O-bound tasks.

managing parallelism manually can be complex and error-prone. This is where joblib excels by simplifying parallel execution.

Using Joblib to Speed Up Your Python Pipelines

Efficient Serialization

from joblib import dump, load

# Saving an object to a file
dump(obj, 'filename.joblib')

# Loading an object from a file
obj = load('filename.joblib')

Parallel Computing

from joblib import Parallel, delayed


def square_number(x):
"""Function to square a number."""
    return x ** 2

numbers = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

# Parallel processing with Joblib
results = Parallel(n_jobs=-1)(delayed(square_number)(num) for num in numbers)

print("Input numbers:", numbers)
print("Squared results:", results)

output

Input numbers: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] Squared results: [1, 4, 9, 16, 25, 36, 49, 64, 81, 100]

Pipeline Integration

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
import joblib

# Load example dataset (Iris dataset)
iris = load_iris()
X = iris.data
y = iris.target

# Split data into training and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Create a pipeline
pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('svc', SVC())
])

# Train the pipeline
pipeline.fit(X_train, y_train)

# Save the pipeline
joblib.dump(pipeline, 'pipeline.joblib')

# Load the pipeline
pipeline = joblib.load('pipeline.joblib')

# Use the loaded pipeline to make predictions
y_pred = pipeline.predict(X_test)

# Evaluate the model
accuracy = accuracy_score(y_test, y_pred)
print(f"Accuracy: {accuracy}")

output

Accuracy: 1.0

Stack in python

Dana — Sat, 12 Aug 2023 17:00:09 +0000

A stack is a linear data structure that follows the principle of Last In First Out (LIFO) This means the last element inserted inside the stack is removed first.

LIFO Principle of Stack:

putting an item on top of the stack is called push.
removing an item is called pop.

Basic Operations of Stack:

Push: Add an element to the top of a stack.
Pop: Remove an element from the top of a stack.
IsEmpty: Check if the stack is empty.
IsFull: Check if the stack is full.
Peek: Get the value of the top element without removing it.

Working of Stack Data Structure:

A pointer called TOP is used to keep track of the top element in the stack.
When initializing the stack, we set its value to -1 so that we can check if the stack is empty by comparing TOP == -1.
On pushing an element, we increase the value of TOP and place the new element in the position pointed to by TOP.
On popping an element, we return the element pointed to by TOP and reduce its value.
Before pushing, we check if the stack is already full.
Before popping, we check if the stack is already empty.

So,

empyty stack => (empty bottle) => top = -1
stack[1] => (push = 1) => top = 0
stack[2] => (push = 2) => top = 1
stack[3] => (push = 3) => top = 2
return stack[2] => (pop = 3) => top = 1

Stack Implementations in Python

"""Stack code."""


def create_stack():
    """Creating a stack."""
    stack = []
    return stack


def check_empty(stack):
    """Creating an empty stack."""
    return len(stack) == 0


def push(stack, item):
    """Adding items into the stack."""
    stack.append(item)
    print("Pushing item " + item)


def pop(stack):
    """Removing an element from the stack."""
    if check_empty(stack):
        """Check for empty stack."""
        return "stack is empty"

    return stack.pop()


stack = create_stack()
push(stack, str(1))
push(stack, str(2))
push(stack, str(3))
push(stack, str(4))
push(stack, str(5))

print("popped item: " + pop(stack))
print("stack after popping an elements " + str(stack))

the output

Pushing item 1
Pushing item 2
Pushing item 3
Pushing item 4
Pushing item 5
popped item: 5
stack after popping an elements ['1', '2', '3', '4']

modular AI

Dana — Wed, 17 May 2023 02:30:03 +0000

في كتير من الاوقات فكرنا في اننا نعمل تطبيقات أسرع وأكثر كفاءة من لغات البرمجه الي احنا عرفنها وشايفين ان لغاتنا غير قادرة على تلبية متطلبتنا وهنا بقي يجي دور لغه برمجه جديده وهي
Mojoprogramminglanguage

موجو هي لغة برمجة جديدة تم تطويرها بواسطة وحدات شركة متخصصة في البنية التحتية للذكاء الاصطناعي. ومصممه خصيصًا لتمكين المطورين من إنشاء تطبيقات أسرع وأكثر قوة للعديد من حالات الاستخدام في تطوير الذكاء الاصطناعي.

لغه موجو هي لغه برمجة تجمع بين استخدام بايثون و سرعه وكفاءه لغه سي وهي لغة تسمح للمطورين بإنشاء تطبيقات عالية الأداء دون الحاجة إلى تعلم لغات وصف الأجهزة القياسية

وعلشان كدا تم إنشاء موجو ، وهي من المفترض أن تكون متوافقًا تمامًا مع نظام بايثون وتوفير سرعة على مستوى لغع سي

https://www.modular.com/mojo

Self Organizing Maps (SOMs)

Dana — Wed, 08 Mar 2023 15:19:44 +0000

Self Organizing Map or Kohonen Map or SOM is a type of Artificial Neural Network which is also inspired by biological models of neural systems.
It follows an unsupervised learning approach and trained its network through a competitive learning algorithm.
SOM is used for clustering and mapping (dimensionality reduction) techniques to map multidimensional data onto lower-dimensional which allows people to reduce complex problems for easy interpretation.
SOM has two layers one is the Input layer and the other one is the Output layer.

How do SOM works?

wij = wij(old) + alpha(t) * (xik - wij(old))

alpha is a learning rate at time t
j denotes the winning vector
i denotes the ith feature of training example
k denotes the kth training example from the input data

After training the SOM network, trained weights are used for clustering new examples. A new example falls in the cluster of winning vectors.

Train the algorithm

Initialize the weights wij random value may be assumed. Initialize the learning rate α.
Calculate squared Euclidean distance.

D(j) = Σ (wij – xi)^2
- i=1 to n
- j=1 to m
Find index J, when D(j) is minimum that will be considered as winning index.
For each j within a specific neighborhood of j and for all i, calculate the new weight.

wij(new)=wij(old) + α[xi – wij(old)]
Update the learning rule

α(t+1) = 0.5 * t
Test the Stopping Condition.

Deep Learning

Dana — Mon, 06 Mar 2023 21:52:53 +0000

Deep learning is a subset of machine learning, which is essentially a neural network with three or more layers.

models are capable enough to focus on the accurate features themselves by requiring a little guidance from the programmer and are very helpful in solving out the problem of dimensionality. Deep learning algorithms are used, especially when we have a huge no of inputs and outputs.

Deep learning is implemented with the help of Neural Networks, and the idea behind the motivation of Neural Network is the biological neurons, which is nothing but a brain cell.

Deep learning is a collection of statistical techniques of machine learning for learning feature hierarchies that are actually based on artificial neural networks.

So basically, deep learning is implemented by the help of deep networks, which are nothing but neural networks with multiple hidden layers.

Architectures:

Deep Neural Networks (DNN)

It is a neural network that incorporates the complexity of certain level, which means several numbers of hidden layers are encompassed in between the input and output layers.

Deep Belief Networks (DBN)

A deep belief network is a class of Deep Neural Network that comprises of multi-layer belief networks.

Recurrent Neural Networks (RNN)

It permits parallel as well as sequential computation and it is exactly similar to that of the human brain.

Accuracy Of Classification

Dana — Tue, 13 Dec 2022 20:21:32 +0000

Accuracy is one metric for evaluating classification models and the fraction of predictions our model got right.

For binary classification, accuracy can also be calculated in terms of positives and negatives as follows:

Where TP = True Positives, TN = True Negatives, FP = False Positives, and FN = False Negatives.

gyroscope

Dana — Tue, 06 Sep 2022 14:35:26 +0000

what is gyroscope?
A gyroscope is a device used for measuring or maintaining orientation and angular velocity.

It is a spinning wheel or disc in which the axis of rotation (spin axis) is free to assume any orientation by itself.

When rotating, the orientation of this axis is unaffected by tilting or rotation of the mounting, according to the conservation of angular momentum.

We now showing A gyroscope in operation, showing the freedom of rotation in all three axes. The rotor will maintain its spin axis direction regardless of the orientation of the outer frame.