DEV Community: Abdellah Hallou

Building and Publishing tree2fs: A Python Package for Folders/Files Generation

Abdellah Hallou — Fri, 21 Nov 2025 00:16:54 +0000

The Problem

Every developer has been there: you're reading documentation, see a beautiful project structure, and think "I'll just recreate this quickly."

Fifteen minutes later, you're still typing mkdir and touch, you've made three typos, and you're wondering why this isn't automated.

I was working on a project when I realized I'd recreated the same project structure three times in one week (I don't use LLMs when I code). That was the moment I decided to build tree2fs.

What is tree2fs?

tree2fs is a Python package that converts tree-formatted text into actual filesystem structures.

Instead of manually creating each folder and file, you write:

project/
├── src/
│   ├── main.py
│   └── utils.py
├── tests/
└── README.md

And run:

tree2fs tree.txt

Done. Your entire structure is created.

Architecture Decisions

1. Two-Phase Architecture

Separate Parsing from Filesystem Operations

Text File → Parser → Tree Data Structure → Builder → Filesystem

2. Tree Data Structure

Use Proper Tree Nodes Instead of Path Stacks

class Node:    
    def __init__(self, data: FileItem, parent: Optional['Node'] = None):
        self.data = data
        self.children: List['Node'] = []
        self.parent = parent

Why:

Clarity: Tree operations are explicit and understandable
CS Fundamentals: Standard tree algorithms (DFS, BFS) work naturally
Extensibility: Easy to add tree operations (find, count, height, etc.)
Debugging: Can inspect tree structure independently
Parent References: Node stores parent, enabling upward traversal

3. Text Format Choice

Tree-Formatted Text Over JSON/TOML/YAML (I may add them too in future versions)

project/
├── src/
│   └── main.py
└── tests/

Why:

Copy-Paste Ready: Most documentation already uses this format
Visual Clarity: Hierarchy is immediately obvious
Source Matching: ChatGPT, Claude, docs all output trees
Natural Comments: ├── config.py # handles configuration
Zero Conversion: No need to translate from other formats Trade-off:
Parsing │ ├ └ ─ is harder than json.loads()

4. Zero Dependencies

Security: No third-party vulnerabilities
Compatibility: Works everywhere Python works
Installation Speed: No dependency resolution
Maintenance: No dependency updates to track
Size: Lightweight package

5. CLI + API Design

Provide Both Interfaces
CLI:

tree2fs tree.txt --verbose --dry-run

Python API:

from tree2fs import create_from_tree

create_from_tree("tree.txt", verbose=True)

Why Both:

CLI: Quick usage, shell scripts, CI/CD
API: Programmatic control, testing, integration
Different Users: Command-line users vs. Python developers ### 6. Dry-Run Mode Add Preview Without Creation

tree2fs tree.txt --dry-run

Why:

Safety: Preview before making changes
Verification: Check if structure is correct
Testing: Verify parser without side effects
User Confidence: See what will happen

Architecture Diagram

┌─────────────────────────────────────────────────────────────┐
│                         tree2fs                             │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  ┌──────────┐     ┌──────────┐      ┌───────────────────┐   │
│  │   CLI    │────▶│   API    │────▶│  create_from_tree │   │
│  └──────────┘     └──────────┘      └─────────┬─────────┘   │
│                                               │             │
│  ┌────────────────────────────────────────────▼─────────┐   │
│  │                   TreeParser                         │   │
│  │  • parse_line(line) → FileItem                       │   │
│  │  • build_tree(file) → Node                           │   │
│  └────────────────────────────┬─────────────────────────┘   │
│                               │                             │
│                               ▼                             │
│  ┌─────────────────────────────────────────────────────┐    │
│  │              Tree Data Structure                    │    │
│  │                                                     │    │
│  │       Node                     FileItem             │    │
│  │    ┌──────────────┐         ┌──────────────┐        │    │
│  │    │ data         │◀───────│ filename     │        │    │
│  │    │ children[]   │         │ level        │        │    │
│  │    │ parent       │         │ comment      │        │    │
│  │    └──────────────┘         └──────────────┘        │    │
│  └────────────────────────────┬────────────────────────┘    │
│                               │                             │
│                               ▼                             │
│  ┌─────────────────────────────────────────────────────┐    │
│  │              FilesystemBuilder                      │    │
│  │  • build(root) → create structure                   │    │
│  │  • _traverse_and_create(node) → DFS                 │    │
│  │  • _create_directory(path)                          │    │
│  │  • _create_file(path)                               │    │
│  └─────────────────────────────┬───────────────────────┘    │
│                                │                            │
│                                ▼                            │
│  ┌─────────────────────────────────────────────────────┐    │
│  │                   Filesystem                        │    │
│  │            📁 Directories & 📄 Files               │    │
│  └─────────────────────────────────────────────────────┘    │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Try It Out

pip install tree2fs

GitHub: https://github.com/ABDELLAH-Hallou/tree2fs
Let me know what you think in the comments!

How I Built an AI Photo Enhancer That Makes Your Selfies Less Tragic (Using Google Gemini & Python)

Abdellah Hallou — Tue, 19 Aug 2025 22:27:51 +0000

Or: "The time I convinced an AI to make my vacation photos look like I actually know how to use a camera"

Hey there, fellow code warriors and photography disasters! 👋

Remember the last time you took a "perfect" photo, only to check it later and wonder why your smile looked awkward and your pose was just... off? Yeah, me too. That's exactly why I built PhotoPro.

The Problem:

Here’s the thing: I’m not a professional photographer. And photo editing? Even worse. Because to “properly” edit photos you apparently need to know:

What filters and sliders actually do
When to use brightness vs. exposure vs. gamma (yes, that’s a thing)
Color theory (whatever that is)

Guess what? I know none of that. And to make things worse, I’m lazy and I don’t want to spend 45 minutes sliding things left and right like I’m on Tinder.

Traditional photo editing software isn’t trivial. Plus, who has time to learn 47 different metrics and filters when you could be doing literally anything else?

So I asked myself:

👉 What if I just told an AI what I want, and let it figure out the math?

Solution: Let AI Do the Heavy Lifting

Instead of learning how to edit, I built an app that lets you upload a photo and describe what you want in plain English (or, y’know, chaotic keyboard smashes), and Google Gemini AI handles the rest.

Want it cinematic? Just say so.
Want it to look like Wes Anderson shot it? Easy.
"Fix my face. Please. I beg you." ? Done.

But — and this is important — I didn’t want to alienate actual photo editors. So PhotoPro also supports good ol’ fashioned filters, grouped in categories for the pros who still like their sliders.

Why Gemini-2.5-flash-image-preview?

Now, here’s the deal:

When we say prompts, we say LLMs.
When we say open source project, we say open source VLMs.

BUT… running open source image models locally requires GPUs that cost more than my monthly rent. So, as a poor engineer working nights and weekends, I took the practical route: use the free Google Gemini Flash API and build the app around it.

No infra headaches, no cloud bills, just free Nano Banana sauce on my images.

The model's capabilities are a major draw, but its accessibility via a free, low-latency API makes it an incredibly attractive option for developers and hobbyists who want to create powerful applications without the significant financial burden of running their own infrastructure.

The Three Enhancement Modes

1. Prompt-only (The Lazy Mode)

Upload your tragic photo → write something like:

“Make this look like it was shot on film in the 80s with moody vibes”

...and boom. Enhanced.

(But you’ll need some prompt-engineering magic — yes, being vague like “make me hotter” doesn’t always work 🙃).

2. Filter-only

Here’s where I let the pros flex their muscles.

I created categories of filters like:

self.basic_filters = ['brightness', 'contrast', 'saturation', 'exposure', 'sharpness']
self.color_filters = ['temperature_tint', 'hsl', 'split_toning', 'curves']
self.artistic_filters = ['vintage', 'cinematic', 'black_white', 'mood_based', 'instagram_presets']
self.effects_filters = ['vignette', 'grain_noise', 'blur', 'light_leaks_flares', 'glitch_pixelate_sketch']
self.ai_filters = ['auto_enhance', 'sky_replacement', 'background_removal_blur', 'face_retouch', 'object_removal', 'style_transfer']
self.editing_filters = ['crop_rotate', 'flip_mirror']
self.overlay_filters = ['add_text', 'stickers_emojis', 'brush_draw', 'frames_borders']

self.filter_categories = {
    "📊 Basic Adjustments": self.basic_filters,
    "🎨 Color & Tone": self.color_filters,
    "🎭 Artistic Styles": self.artistic_filters,
    "✨ Visual Effects": self.effects_filters,
    "🤖 AI-Powered": self.ai_filters,
    "📐 Transform & Edit": self.editing_filters,
    "📝 Overlays & Text": self.overlay_filters
}

⚠️ Full honesty here: I didn’t invent these categories myself.

I literally asked ChatGPT to generate them for me, because — as I said — I suck at photo editing.

If I had to come up with this list, I’d have stopped at “brightness” and “make me look cool.”

The app flow is:

Choose filters from categories (checkboxes).
Configure them with sliders/inputs.
Get a structured prompt + apply.

3. Prompt + Filters

The best of both worlds.

A text prompt ("Make me look like a cyberpunk protagonist").
Manual filters ("cinematic")

PhotoPro eats the combo prompt + filters, and outputs something magical.

🛠️ Under the Hood

Frontend: Streamlit (fast prototyping, zero effort UI)
Backend: Google Gemini Flash API (free tier = lifesaver)
Image Processing: PIL + NumPy (still useful for prep & save)
Logic: Python

Try It Yourself!

Want to see the magic?

👤 For Users

Live Demo (requires a Gemini API key: https://aistudio.google.com/app/apikey)

💻 For Developers

Clone the Repo

git clone https://github.com/ABDELLAH-Hallou/gemini-photo-filters
cd gemini-photo-filters

Set Up Your Environment

# Create a virtual environment (because dependency hell is real)
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate
# Install the magic
pip install -r requirements.txt

Get Your API Key

Go to Google AI Studio
Create an API key (it's free-ish)
Add it to .streamlit/secrets.toml:

GEMINI_API_KEY = "your-actual-api-key-here"

Run the Thing

streamlit run app.py

Then navigate to localhost:8501 and start making your photos less tragic!

You can always customize filters in utils/filters.py

What's Next? (The Roadmap of Dreams)

Here's what I'm planning to add when I'm not busy procrastinating:

Style Transfer: Train the AI on your preferred aesthetic.
Social Media Integration: Auto-post your enhanced photos (with your permission, obviously).
Mobile App: Because who enhances photos on desktop anymore?
Collaborative Features: Share your enhancement settings with friends.
AI Roast Mode: Let the AI critique your photos (for the masochists).

Final Thoughts

Building PhotoPro taught me that sometimes the best solutions come from solving your own problems. I needed better photos, AI existed, and Python made it possible to connect the two.

🛠️ Setting Up Airflow with Celery, RabbitMQ, and PostgreSQL — Solving Real-World Integration Issues

Abdellah Hallou — Fri, 06 Jun 2025 01:10:40 +0000

Many data engineers struggle to integrate various tools like Airflow, Celery, RabbitMQ, and PostgreSQL into a cohesive architecture. I’ve been there. While online tutorials helped me configure each tool individually, they rarely covered the practical integration challenges that inevitably arise.

That’s exactly why I wrote this article: to consolidate all the resources I used, fill in the missing pieces, and provide a working setup with common errors resolved — based on real experience.

Let’s get our hands dirty. But don’t worry — baby steps first 🍼

⚠️ Before We Start: A Quick Overview of the Stack

I won’t dive deep into each tool’s internal workings, just quick and simple definitions to get you on board:

Airflow: A task scheduler for defining and executing workflows. By default, it uses the SequentialExecutor and SQLite.

PostgreSQL: Recommended for production-grade Airflow setups instead of SQLite.

Celery: A distributed task queue that handles asynchronous job execution. Required if you’re using CeleryExecutor in Airflow.

RabbitMQ: A message broker used by Celery. While Redis is the default, we’ll be using RabbitMQ here.

🚀 Step-by-Step Setup

1. 🔧 Install Airflow and Essential Libraries

pip3 install apache-airflow[gcp,sentry,statsd]

2. 🗄️ Initialize Airflow Database

Navigate to the airflow directory and initialize the airflow database:

cd airflow
airflow db init

After initialization, you'll notice new files and directories in the airflow folder. Create a dags folder where all future DAGs will be stored and accessed by Airflow components:

mkdir dags

3. 👤 Create an Airflow Admin User

airflow users create --username admin --password your_password --firstname your_first_name --lastname your_last_name --role Admin --email your_email@domain.com

Verify the user was created successfully:

airflow users list

Expected output:

id | username | email                 | first_name      | last_name      | roles
===+==========+=======================+=================+================+======
 1 | admin    | your_email@domain.com | your_first_name | your_last_name | Admin

🎉 You’re now an admin. Feels good, doesn’t it?

4. ▶️ Start Scheduler and Webserver (in separate terminals)

Start scheduler (Terminal 1):

airflow scheduler

In a new terminal (Terminal 2), activate the virtual environment and start the webserver:

source airflow_env/bin/activate
cd airflow
airflow webserver

Once both services are initialized, open your browser and navigate to http://localhost:8080/.

Port 8080 is the default port for Airflow. If it's occupied, modify the port number in the airflow.cfg file.

If you see a jungle of random DAGs — yeah, Airflow comes with prebuilt DAGs that we don’t need. Feel free to explore the UI and experiment with these examples to understand how Airflow works.

🧹 Clean Up Example DAGs (Optional)

If you prefer a clean slate without example DAGs, you can disable them by setting the following variable in airflow.cfg file:

AIRFLOW__CORE__LOAD_EXAMPLES=False

Creating Your First DAG

To create your first custom DAG, follow the detailed instructions at:

👉 How to Create First DAG in Airflow – GeeksForGeeks

And yes, once created, you’ll see it in the dashboard!

🧨 Now Airflow Works. You Happy??

Haha… thought that was it? Nah. The real fun begins now. Time to swap out the baby-defaults and give Airflow a proper backend and executor.

🗃️ From SQLite to PostgreSQL: No More Toy Databases.

SQLite got us off the ground, but we’re building an actual system now, we’re going PostgreSQL + LocalExecutor.

1. Install PostgreSQL

sudo apt install postgresql

2. Configure PostgreSQL for Airflow

Access the PostgreSQL shell and create the necessary database and user:

sudo -u postgres psql

Inside the prompt:

CREATE DATABASE airflow_db;
CREATE USER airflow_user WITH PASSWORD 'airflow_pass';
GRANT ALL PRIVILEGES ON DATABASE airflow_db TO airflow_user;

🗂️ Update Airflow’s Config File

Update your airflow.cfg file with the following changes:

Change the executor:
```
executor = LocalExecutor
```

Update the database connection:

sql_alchemy_conn = postgresql+psycopg2://airflow_user:airflow_pass@localhost:5432/airflow_db

If you created a new PostgreSQL user for Airflow, the default search_path should work without changes. However, if you're using an existing user with a custom search_path, update it:

ALTER USER airflow_user SET search_path = public;

📜 If Required: Update `pg_hba.conf` (Because Sometimes It Just Doesn’t Work)

If needed, add the Airflow user to PostgreSQL's access control list:

Open the pg_hba.conf file (typically in the PostgreSQL data directory)
Locate the “Local connections” or “IPv4 local connections” section (depending on the version of Postgres you are using)
Add the following line:

host    all             airflow_user             127.0.0.1/32            md5

Then restart PostgreSQL:

sudo service postgresql restart

4. Re-initialize Airflow DB

Initialize your Airflow database with the new PostgreSQL backend:

airflow db init

Takes a few seconds, so… stretch, hydrate, or scream internally.

🛡️ Optional: Fix Permissions (Because PostgreSQL Likes Drama)

If Airflow starts whining about permissions, you start by:

Checking your PostgreSQL users:

SELECT usename FROM pg_user;

To grant privileges, you can either:

🔒 Keep it minimal:

GRANT CREATE, USAGE ON SCHEMA public TO airflow_user;

💥 Or go nuclear:

ALTER ROLE airflow_user WITH SUPERUSER;

(Use superuser only if you're lazy...)

👨‍💻 Recreate Admin User (Now for PostgreSQL Setup)

airflow users create \
  --username admin \
  --firstname FirstName \
  --lastname LastName \
  --role Admin \
  --email dummy@xyz.com \
  --password admin
# (The Fancy Way)

Boom. You’re the boss now.

✅ Run Airflow Again (Like a Legend)

In terminal 1:

airflow webserver

In terminal 2:

airflow scheduler

Now open http://localhost:8080 and enjoy the view 😌

Now Airflow is running with PostgreSQL and ready for Celery integration. Happy? Haha, now the real fun begins…

In the previous section, you mastered the Local Executor and achieved parallel task execution in Airflow. Now it's time to level up with the Celery Executor to build truly production-ready pipelines. But first, we need to set up our supporting cast: Celery, Flower (the web server for Celery), and RabbitMQ as our message broker.

Deep breath - Here we go!

🐰 Setting Up RabbitMQ — Your Message Broker

RabbitMQ is like the post office of your pipeline — receiving, queueing, and delivering messages between your Airflow components. Built with Erlang, it’s designed to handle large-scale message traffic with resilience.

📦 1. Install Erlang

First, we need Erlang (because apparently rabbits speak Erlang):

sudo apt-get install erlang

Perfect time to grab a cup of mint tea, while the installation does its magic... ☕.

🐇 2. Install RabbitMQ Server

sudo apt-get install rabbitmq-server

Celebrate with a second cup, this time atay b chiba.

▶️ 3. Start the RabbitMQ Server

Let’s boot it up and verify it’s alive:

sudo service rabbitmq-server start

service --status-all

You should see rabbitmq-server running happily as a service. If it's not running, give it another gentle nudge!

🌐 4. Enable RabbitMQ Management Dashboard

RabbitMQ comes with a sleek UI. Let’s unlock it:

sudo rabbitmq-plugins enable rabbitmq_management

The default listening port for RabbitMQ is 15672 (memorize this number, you'll be seeing it a lot!)
Navigate to http://localhost:15672/ for the dashboard
Default credentials: username "guest" with password "guest" (security level: toddler-proof)
Voilà! A dashboard to spy on your queues.

🔐 5. Create a Custom User and Virtual Host

Let’s create a user and a dedicated vhost for Airflow:

sudo rabbitmqctl add_user admin admin
sudo rabbitmqctl rabbitmqctl add_vhost airflowvhost

Make our "admin" user the boss:

sudo rabbitmqctl set_user_tags admin administrator

Grant permissions:

sudo rabbitmqctl set_permissions -p airflowvhost admin "." "." ". "

Now you can log in with the "admin" credentials and feel like you actually know what you're doing!

🌱 Installing Celery & Flower

⚙️ 1. Install Celery and Flower

Celery is your task executor, and Flower is your Celery UI garden 🌸
Install both:

sudo pip3 install apache-airflow[celery]

This will also pull in required dependencies. (Time for more Moroccan tea?)

🌼 2. Launch the Flower Dashboard

Run Flower with:

airflow celery flower

Default port: 5555 (another number for your collection!)
Visit http://localhost:5555/ to see Flower in action
If you see a clean interface, congratulations! If not... well, that's what the troubleshooting section is for 😅

🛠️ Airflow Configuration — Connect the Dots

Now for the moment of truth - let's tell Airflow about its new friends. Open your airflow.cfg file and update these parameters:

executor = CeleryExecutor
[celery]
broker_url = amqp://admin:admin@localhost:5672/airflowvhost # By default, RabbitMQ will listen on port 5672 on all available interfaces
celery_result_backend  = db+postgresql+psycopg2://admin:admin@localhost:5432/airflow_db # Port 5432 is dedicated to PostgreSQL connections
broker_connection_retry_on_startup = True

💡 To avoid task revocation due to idle timeouts, set the consumer timeout in either:

RabbitMQ config (/etc/rabbitmq/rabbitmq.conf):

consumer_timeout = 31622400000

or directly in airflow.cfg:

[celery_broker_transport_options]
consumer_timeout = 31622400000

🔄 Restart Services

Make sure all gears are turning:

Restart PostgreSQL:

sudo service postgresql restart

Start RabbitMQ:

sudo service rabbitmq-server start

Initialize the Airflow DB:

airflow db init

✅ Final Checklist: System Up & Running

Time to see if all your hard work pays off! You'll need four terminals (yes, four - you're basically a command line DJ now 🎧):

Component	Command	URL
Webserver	`airflow webserver`	localhost:8080
Scheduler	`airflow scheduler`	–
Celery Worker	`airflow celery worker`	–
Flower UI	`airflow celery flower`	localhost:5555
RabbitMQ Dashboard	–	localhost:15672

Don’t forget to clean up old DAGs from the UI to let the scheduler re-scan everything afresh.

& That’s it.

📝 Bonus Tip

Always keep an eye on the celery.log file.. it's your best friend when things go sideways (and they occasionally will, because that's just how distributed systems roll 🤷‍♂️).

Reference:

https://habr.com/ru/companies/neoflex/articles/736292/

https://medium.com/accredian/setting-up-apache-airflow-in-ubuntu-324cfcee1427

https://documentation.ubuntu.com/server/how-to/databases/install-postgresql/index.html

https://medium.com/plumbersofdatascience/from-sqlite-to-postgresql-a-lighthearted-guide-to-upgrading-your-airflow-database-7b9c8de961b5

https://medium.com/coding-blocks/creating-user-database-and-adding-access-on-postgresql-8bfcd2f4a91e

https://stackoverflow.com/questions/69828547/precondition-failed-delivery-acknowledge-timeout-on-celery-rabbitmq-with-geve

https://airflow.apache.org/docs/apache-airflow/stable/howto/set-up-database.html

https://stackoverflow.com/questions/74390647/postgres-airflow-db-permission-denied-for-schema-public

https://stackoverflow.com/questions/10757431/postgres-upgrade-a-user-to-be-a-superuser

Solving pandas pickle compatibility issues across different versions

Abdellah Hallou — Sun, 16 Mar 2025 18:59:23 +0000

Have you ever encountered the frustrating error when trying to read a pickle file created with a different version of pandas? You're not alone. This common issue affects many data scientists and developers working in collaborative environments or maintaining long-term projects. Let's explore how to solve this problem effectively.

When you save a pandas DataFrame using to_pickle(), the serialization process is specific to the pandas version used. This means that pickle files created with newer versions of pandas may not be readable by older versions, leading to compatibility errors.

Solution Options

Option 1: Use the Same Version (Simple but Limited)

The most straightforward solution is to ensure you're using the same version (or a later one) of pandas as the one used to create the pickle file. However, this isn't always practical, especially in team environments or when working with legacy systems.

Option 2: Convert to CSV (Universal but Limited)

For simple dataframes without complex objects, converting to CSV offers excellent compatibility:

# With newer pandas version
import pandas as pd
data = pd.read_pickle('path/to/file.pkl')
data.to_csv('path/to/file.csv', index=False)

# With older pandas version
data = pd.read_csv('path/to/file.csv')

This approach works well for most tabular data but has limitations when dealing with complex data types.

Option 3: Use HDF Format (Best for Complex Data)

For dataframes containing objects like lists and arrays in individual cells, the HDF format provides better compatibility:

Step 1: Load data with the newest version of Python and pandas

import pandas as pd
import pickle
data = pd.read_pickle('path/to/file.pkl')

Step 2: Save as HDF with protocol 4

pickle.HIGHEST_PROTOCOL = 4
data.to_hdf('output/folder/path/to/file.hdf', 'df')

You may need to install the required dependency:

pip install tables

👉🏻Understanding Pickle Protocols:
Python's pickle module has several protocol versions:

Protocol 0: The original ASCII-based protocol. Human-readable but inefficient for binary data.

Protocol 1: Binary format, introduced in Python 2.3. More efficient than Protocol 0.

Protocol 2: Added in Python 2.3. Supports more efficient pickling of classes and instances.

Protocol 3: Default in Python 3.0 - 3.7. More efficient for new-style classes.

Protocol 4: Introduced in Python 3.4. Supports larger objects and more efficient storage of binary data.

Protocol 5: Added in Python 3.8. Optimized for out-of-band data and better handling of certain objects.

When you save a pandas DataFrame using to_pickle(), pandas uses Python's pickle module under the hood. The default behavior of pandas is to use the highest available pickle protocol version in your Python environment.

Step 3: Load the HDF file with the older python version

import pandas as pd
data = pd.read_hdf('path/to/file.hdf')

While pickle files offer convenience for pandas users, their version-specific nature can create headaches. By understanding the options for cross-version compatibility, you can choose the right approach for your specific data needs. When in doubt, the HDF format provides an excellent balance of compatibility and data integrity for complex pandas DataFrames.

⚠️Remember: data interchange formats are a crucial but often overlooked aspect of data science workflows. Taking the time to implement proper serialization strategies can save hours of debugging and data reconstruction later.

From Data to Deployment

Abdellah Hallou — Mon, 28 Oct 2024 12:22:20 +0000

DataWhisper: Mastering DL Project Lifecycle

Author: Abdellah Hallou (LinkedIn, Twitter)

Welcome to the Deep Learning Project Starter Guide! This tutorial serves as a comprehensive resource for anyone looking to dive into the exciting world of deep learning. Whether you're a beginner or an experienced developer, this guide will take you through the process of building a deep learning project from start to finish.

What you'll learn
Who should follow this tutorial
Need help or have questions?
Let's get started!
Imports and loading the dataset
Dataset Structure
Exploratory Data Analysis (EDA)
Preprocess the data
Build the model
Evaluate accuracy
Save and Export the Model
Make predictions
Deployment
- Create a new flutter project
- Configuring the Camera
- Creating the Camera Screen
- Integrating Image Upload
- Object Recognition with TensorFlow Lite
- Running the Model on Images
- Displaying Results in a Dialog
- Building the User Interface

What you'll learn

In this tutorial, you will learn the essential steps involved in creating and deploying a deep-learning model in a mobile app. We will cover the following topics:

Preparing the data: We'll explore various methods for data preprocessing to ensure a robust and reliable dataset for training.
Model creation: You'll discover how to design and build your CNN model.
Training the model: We'll delve into the process of training your deep learning model using TensorFlow.
Deployment in a mobile app: Once your model is trained, we'll guide you through the steps to integrate it into a mobile app using TensorFlow Lite. You'll understand how to make predictions on the go!

Who should follow this tutorial

This tutorial is suitable for beginners and intermediate developers with a basic understanding of deep learning concepts and Python programming. Whether you're a data scientist, machine learning enthusiast, or mobile app developer, this guide will equip you with the necessary knowledge to kick-start your deep learning project.

Need help or have questions?

If you encounter any issues, have questions, or need further clarification while following this tutorial, don't hesitate to create a GitHub issue in this repository From-Data-to-Deployment. I'll be more than happy to assist you and provide the necessary guidance.

To create an issue, click on the "Issues" tab at the top of this repository's page and click the "New issue" button. Please provide as much context and detail as possible about the problem you're facing or the question you have. This will help me understand your concern better and provide you with a prompt and accurate response.

Your feedback is valuable and can help improve this tutorial for other users as well. So, don't hesitate to reach out if you need any assistance. Let's learn and grow together!

Let's get started!

To start, ensure you have the required dependencies and libraries installed. The tutorial is divided into easy-to-follow sections, each covering a specific aspect of the deep learning project workflow. Feel free to jump to the sections that interest you the most or follow along from beginning to end.

Are you ready?

Imports and loading the dataset

Let's start the necessary imports for our code. We will use the Fashion Mnist dataset in this tutorial.

# Import the necessary libraries
from __future__ import print_function
import keras
from google.colab import drive
import os
import numpy as np
from keras.models import Sequential
from keras.layers import Dense, Dropout, Activation, Flatten, BatchNormalization
from keras.layers import Conv2D, MaxPooling2D
from keras.wrappers.scikit_learn import KerasClassifier
from keras import backend as K
from sklearn.model_selection import GridSearchCV
import tensorflow as tf
from keras.utils.vis_utils import plot_model
import matplotlib.pyplot as plt

Dataset Structure

In any deep learning project, understanding the data is crucial. Before diving into model creation and training, let's start by loading the data and gaining insights into its structure, variables, and overall characteristics.

# Load the Fashion MNIST dataset
fashion_mnist = tf.keras.datasets.fashion_mnist
(x_train, y_train), (x_test, y_test) = fashion_mnist.load_data()

Exploratory Data Analysis (EDA)

Now that the data is loaded, let's perform some exploratory data analysis to gain a better understanding of its characteristics.

print("Shape of the training data : ",x_train.shape)
print("Shape of the testing data : ",x_test.shape)

Shape of the training data :  (60000, 28, 28)
Shape of the testing data :  (10000, 28, 28)

The Fashion MNIST dataset contains 70,000 grayscale images in 10 categories. The images show individual articles of clothing at low resolution (28 by 28 pixels), as seen here:

60,000 images are used to train the network and 10,000 images to evaluate how accurately the network learned to classify images.

# Printing unique values in training data
unique_labels = np.unique(y_train, axis=0)
print("Unique labels in training data:", unique_labels)

Unique labels in training data: [0 1 2 3 4 5 6 7 8 9]

The labels are an array of integers, ranging from 0 to 9. These correspond to the class of clothing the image represents:
| Label | RClass |
| - |-|
| 0 | T-shirt/top|
| 1 | Trouser|
| 2 |Pullover|
| 3 |Dress|
| 4 |Coat|
| 5 |Sandal|
| 6 |Shirt|
| 7 |Sneaker |
| 8 |Bag|
| 9 | Ankle boot |

Since the class names are not included with the dataset, store them here to use later when plotting the images:

# Numeric labels
numeric_labels = np.sort(np.unique(y_train, axis=0))
# String labels
string_labels = np.array(['T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat','Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle boot'])
# Mapping numeric labels to string labels
numeric_to_string = dict(zip(numeric_labels, string_labels))
print("Numeric to String Label Mapping:")
print(numeric_to_string)

Numeric to String Label Mapping:
{0: 'T-shirt/top', 1: 'Trouser', 2: 'Pullover', 3: 'Dress', 4: 'Coat', 5: 'Sandal', 6: 'Shirt', 7: 'Sneaker', 8: 'Bag', 9: 'Ankle boot'}

Preprocess the data

The data must be preprocessed before training the network.
We start by defining the number of classes in our dataset (which is 10 in this case) and the dimensions of the input images (28x28 pixels).

num_classes = 10
# input image dimensions
img_rows, img_cols = 28, 28

This part is responsible for reshaping the input image data to match the expected format for the neural network model. The format depends on the backend being used (e.g., TensorFlow or Theano). In this snippet, we check the image data format using K.image_data_format() and apply the appropriate reshaping based on the result.

if K.image_data_format() == 'channels_first':
    x_train = x_train.reshape(x_train.shape[0], 1, img_rows, img_cols)
    x_test = x_test.reshape(x_test.shape[0], 1, img_rows, img_cols)
    input_shape = (1, img_rows, img_cols)
else:
    x_train = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1)
    x_test = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1)
    input_shape = (img_rows, img_cols, 1)

The pixel values of the images in the data fall within the range of 0 to 255.
Scale these values to a range of 0 to 1 before feeding them to the CNN model.

x_train = x_train.astype('float32')
x_test = x_test.astype('float32')
x_train /= 255
x_test /= 255

Convert the class labels (represented as integers) to a binary class matrix format, which is required for multi-class classification problems.

# convert class vectors to binary class matrices
y_train = keras.utils.to_categorical(y_train, num_classes)
y_test = keras.utils.to_categorical(y_test, num_classes)

Build the model

In this step, we define and build a convolutional neural network (CNN) model for image classification. The model architecture consists of multiple layers such as convolutional, pooling, dropout, and dense layers. The build_model function takes the number of classes, training and testing data as input and returns the training history and the built model.

def build_model(num_classes,x_train,y_train,x_test,y_test):
  model = Sequential()
  model.add(BatchNormalization())
  model.add(Conv2D(64, kernel_size=(4, 4), padding='same',
                  activation='relu',
                  input_shape=input_shape))
  model.add(MaxPooling2D(pool_size=(2, 2)))
  model.add(Dropout(0.1))
  model.add(Conv2D(64, kernel_size=(4, 4),
                  activation='relu',
                  input_shape=input_shape))
  model.add(MaxPooling2D(pool_size=(2, 2)))
  model.add(Dropout(0.3))
  model.add(Flatten())
  model.add(Dense(256, activation='relu'))
  model.add(Dropout(0.5))
  model.add(Dense(64, activation='relu'))
  model.add(BatchNormalization())
  model.add(Dense(num_classes, activation='softmax'))

  model.compile(loss=keras.losses.categorical_crossentropy,
                optimizer=tf.optimizers.Adam(),
                metrics=['accuracy'])
  history = model.fit(x_train, y_train,epochs=80,batch_size=250,validation_data=(x_test, y_test))
  return history,model

history,model = build_model(num_classes,x_train,y_train,x_test,y_test)

Epoch 80/80
240/240 [==============================] - 211s 878ms/step - loss: 0.0866 - accuracy: 0.9678 - val_loss: 0.2308 - val_accuracy: 0.9289

# Function: is_drive_mounted
# Description: Checks if the Google Drive is mounted.
# Returns: True if the drive is mounted, False otherwise.
def is_drive_mounted():
    return os.path.isdir('/content/drive')
# Function: mount_drive
# Description: Mounts the Google Drive if it is not already mounted.
# Prints a message if the drive is already mounted.
def mount_drive():
  if not is_drive_mounted():
    drive.mount('/content/drive')
  else:
      print('Drive already mounted.')
# Function: move_to_drive
# Description: Moves a file to the specified folder in Google Drive.
# If the file already exists in the destination folder, prints a message.
# Parameters:
#   - filename: Name of the file to be moved.
#   - folder: Name of the destination folder.
# Returns: The path of the moved file in Google Drive.
def move_to_drive(filename, folder):
  mount_drive()
  # Define the file path and name in your Google Drive
  output_file_path = '/content/drive/MyDrive/'+folder
  if os.path.exists(output_file_path+filename):
    print("The file exists.")
  else:
    # Copy the downloaded file to your Google Drive
    !cp /content/'{filename}' '{output_file_path}'
  return output_file_path+filename
# Function: save_class_names
# Description: Saves a list of class names to a file.
# The file is created in the specified folder.
# Parameters:
#   - folder_path: Path of the destination folder.
#   - filename: Name of the output file.
# Returns: The path of the saved file in Google Drive and the list of class names.
def save_class_names(folder_path, filename):
  class_names = ['T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat','Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle boot']
  with open(filename, 'w') as f:
    for class_name in class_names:
        f.write(class_name + '\n')
  return move_to_drive(filename, folder_path),class_names
# Function: h52tflite
# Description: Converts a Keras model to TFLite format and saves the model.
# Parameters:
#   - model: Keras model to be converted.
#   - filename: Name of the output file.
# Returns: The path of the saved TFLite model in Google Drive and the TFLite model itself.
def h52tflite(model,filename):
  # Convert the model to TFLite format
  converter = tf.lite.TFLiteConverter.from_keras_model(model)
  tflite_model = converter.convert()

  # Save the TFLite model
  with open(filename, 'wb') as f:
      f.write(tflite_model)

  return move_to_drive(filename,'DLTutorial/model/'),tflite_model
# Function: plot_model_evaluation
# Description: Plots the training and validation loss and accuracy of a model.
# Parameters:
#   - model: Model to be evaluated.
#   - history: Training history of the model.
def plot_model_evaluation(model,history):
  plot_model(model, to_file='model_plot.png', show_shapes=True, show_layer_names=True)
  fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 4))

  ax1.plot(history.history['loss'], label = 'training loss')
  ax1.plot(history.history['accuracy'], label = 'training accuracy')
  ax1.legend()
  ax1.set_title('Training')

  ax2.plot(history.history['val_loss'], label = 'validation loss')
  ax2.plot(history.history['val_accuracy'], label = 'validation accuracy')
  ax2.legend()
  ax2.set_title('Validation')

  plt.show()
# Function: export_model
# Description: Saves a Keras model as an .h5 file, converts it to TFLite format, and saves the TFLite model.
# Parameters:
#   - model: Keras model to be exported.
#   - filename: Name of the output files.
# Returns: The paths of the saved model and TFLite model in Google Drive, and the TFLite model itself.
def export_model(model,filename):
  model.save(filename+'.h5')
  model_path = move_to_drive(filename+'.h5', 'DLTutorial/model/')
  tflite_model_path,tflite_model = h52tflite(model,filename+".tflite")
  return model_path,tflite_model_path,tflite_model

Evaluate accuracy

To assess the performance of the trained model, we evaluate it on the test data. The evaluate method is used to calculate the test loss and accuracy. These metrics are then printed to the console.

plot_model_evaluation(model,history)

# Evaluate the model on test data
test_loss, test_accuracy = model.evaluate(x_test, y_test, verbose=0)
print(f"Test Loss: {test_loss:.4f}")
print(f"Test Accuracy: {test_accuracy:.4f}")

Test Loss: 0.2308
Test Accuracy: 0.9289

Save and Export the Model

After training the model, we save it in the Hierarchical Data Format (HDF5) file format using the save method. The model is then exported to the Google Drive by calling the move_to_drive function. Additionally, the model is converted to the TensorFlow Lite format using the h52tflite function, and the resulting TFLite model is also saved in the Google Drive. The paths of the saved model and TFLite model are returned.

model_path,tflite_model_path,tflite_model = export_model(model,"model")

Make predictions

To visualize the model's predictions, we select a random set of test images. The model predicts the class labels for these images using the predict method. The predicted labels are then compared with the ground truth labels to display the images along with their corresponding predicted labels using matplotlib.

# Predict class labels for test images
predictions = model.predict(x_test)
predicted_labels = np.argmax(predictions, axis=1)

# Visualize a random selection of test images along with their predicted labels
num_rows = 5
num_cols = 5
num_images = num_rows * num_cols
random_indices = np.random.choice(len(x_test), size=num_images, replace=False)

fig, axes = plt.subplots(num_rows, num_cols, figsize=(10, 10))
fig.suptitle("Fashion MNIST - Predictions", fontsize=16)
fig.tight_layout(pad=2.0)

for i, index in enumerate(random_indices):
    ax = axes[i // num_cols, i % num_cols]
    ax.imshow(x_test[index], cmap='gray')
    ax.set_title(f"Pred: {string_labels[predicted_labels[index]]}")
    ax.axis('off')

plt.show()

for more information about the model, check these resources:

Deployment

Create a new Flutter project

Before creating new Flutter project, make sure that the Flutter SDK and other Flutter app development-related requirements are properly installed: https://docs.flutter.dev/get-started/install/windows

After the project has been set up, we will implement the UI to allow users to take pictures or upload images from the gallery and perform object recognition using the exported TensorFlow Lite model.
First, we need to install these packages:

camera: 0.10.4
image_picker:
tflite: ^1.1.2

To do so copy the following code snippet and paste it into the pubspec.yaml file of the project:

dependencies:
  camera: 0.10.4
  image_picker:
  tflite: ^1.1.2

Import the necessary packages in the main.dart file of the project

import 'dart:async';
import 'dart:io';
import 'package:flutter/material.dart';
import 'package:camera/camera.dart';
import 'package:image_picker/image_picker.dart';
import 'package:tflite/tflite.dart';

Configuring the Camera

To enable camera functionality, we'll utilize the camera package. First, import the necessary packages and instantiate the camera controller. Use the availableCameras() function to get a list of available cameras. In this tutorial, we'll use the first camera in the list.

void main() async {
  WidgetsFlutterBinding.ensureInitialized();
  final cameras = await availableCameras();
  final firstCamera = cameras.first;
  runApp(MyApp(firstCamera));
}

Creating the Camera Screen

Create a new StatefulWidget called CameraScreen that will handle the camera preview and image capture functionality. In the initState() method, initialize the camera controller and set the resolution preset. Additionally, implement the _takePicture() method, which captures an image using the camera controller.

// main.dart
class CameraScreen extends StatefulWidget {
  final CameraDescription camera;

  const CameraScreen(this.camera);

  @override
  _CameraScreenState createState() => _CameraScreenState();
}

class _CameraScreenState extends State<CameraScreen> {
  late CameraController _controller;
  late Future<void> _initializeControllerFuture = Future.value();

  @override
  void initState() {
    super.initState();
    _initCameraController();
  }

  Future<void> _initCameraController() async {
    _controller = CameraController(
      widget.camera,
      ResolutionPreset.medium,
    );

    _initializeControllerFuture = _controller.initialize();

    setState(() {});
  }
    Future<void> _takePicture() async {
    try {
      await _initializeControllerFuture;
      final image = await _controller.takePicture();
      await _processImage(image.path);
    } catch (e) {
      print('Error capturing image: $e');
    }
  }
  // ...
}

Integrating Image Upload

To allow users to upload images from the gallery, import the image_picker package. Implement the _pickImage() method, which utilizes the ImagePicker class to select an image from the gallery. Once an image is selected, it can be processed using the _processImage() method.

// main.dart
class _CameraScreenState extends State<CameraScreen> {
  // ...

  Future<void> _pickImage() async {
    try {
      final imagePicker = ImagePicker();
      final pickedImage = await imagePicker.getImage(source: ImageSource.gallery);

      if (pickedImage != null) {
        await _processImage(pickedImage.path);
      }
    } catch (e) {
      print('Error picking image: $e');
    }
  }

  // ...
}

Object Recognition with TensorFlow Lite

To perform object recognition, we'll use the TensorFlow Lite. Begin by importing the tflite package. In the _initTensorFlow() method, load the TensorFlow Lite model and labels from the assets. You can specify the model and label file paths and adjust settings like the number of threads and GPU delegate usage.

// main.dart
class _CameraScreenState extends State<CameraScreen> {
  // ...

  @override
  void initState() {
    super.initState();
    _initCameraController();
    _initTensorFlow();
  }

  Future<void> _initTensorFlow() async {
    String? res = await Tflite.loadModel(
      model: "assets/model/model.tflite",
      labels: "assets/model/labels.txt",
      numThreads: 1,
      isAsset: true,
      useGpuDelegate: false,
    );
  }

  // ...
}

Running the Model on Images

Implement the _objectRecognition() method, which takes an image file path as input and runs the TensorFlow Lite model on the image. The method returns the label of the recognized object.

// main.dart
class _CameraScreenState extends State<CameraScreen> {
  // ...
  Future<String> _objectRecognition(String filepath) async {
    var recognitions = await Tflite.runModelOnImage(
      path: filepath,
      numResults: 10,
      threshold: 0.1,
      asynch: true,
    );

    return recognitions![0]["label"].toString();
  }

  // ...
}

Displaying Results in a Dialog

When an image is processed, display the result in a dialog box using the showDialog() method. Customize the dialog to show the recognized object label and provide an option to cancel.

// main.dart
class _CameraScreenState extends State<CameraScreen> {
  // ...
Future<void> _processImage(String imagePath) async {
    final String label = await _objectRecongnition(imagePath);
    showDialog(
      context: context,
      barrierDismissible: false, dialog
      builder: (BuildContext context) {
        return Dialog(
          child: Container(
            padding: EdgeInsets.all(20),
            child: Column(
              mainAxisSize: MainAxisSize.min,
              children: [
                Text(label),
                SizedBox(height: 20),
                ElevatedButton(
                  onPressed: () {
                    Navigator.of(context).pop(); // Close the dialog
                  },
                  child: Text('Cancel'),
                ),
              ],
            ),
          ),
        );
      },
    );
  }
  // ...
}

Building the User Interface

// main.dart
void main() async {
  WidgetsFlutterBinding.ensureInitialized();

  final cameras = await availableCameras();
  final firstCamera = cameras.first;


  runApp(MyApp(firstCamera));
}

class MyApp extends StatelessWidget {
  final CameraDescription camera;

  const MyApp(this.camera);

  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      title: 'Fashion Mnist',
      theme: ThemeData(
        primarySwatch: Colors.blue,
      ),
      home: CameraScreen(camera),
    );
  }
}

class CameraScreen extends StatefulWidget {
  CameraScreen(CameraDescription camera);

  @override
  _CameraScreenState createState() => _CameraScreenState();
}

class _CameraScreenState extends State<CameraScreen> {
  late CameraController _controller;
  late Future<void> _initializeControllerFuture = Future.value();

  @override
  void initState() {
    super.initState();
    _initCameraController();
    _initTensorFlow();
  }
  Future<void> _initCameraController() async {
    final cameras = await availableCameras();
    final firstCamera = cameras.first;
    _controller = CameraController(
      firstCamera,
      ResolutionPreset.medium,
    );
    _initializeControllerFuture = _controller.initialize();
    setState(() {});
  }
  @override
  void dispose() {
    _controller.dispose();
    super.dispose();
  }

  Future<void> _takePicture() async {
    try {
      await _initializeControllerFuture;
      final image = await _controller.takePicture();
      await _processImage(image.path);
    } catch (e) {
      print('Error capturing image: $e');
    }
  }

  Future<void> _pickImage() async {
    try {
      final imagePicker = ImagePicker();
      final pickedImage = await imagePicker.getImage(
          source: ImageSource.gallery);
      if (pickedImage != null) {
        await _processImage(pickedImage.path);
      }
    }catch(e){
      print('Error picking image: $e');
    }
  }

  Future<void> _processImage(String imagePath) async {
    final String label = await _objectRecongnition(imagePath);
    showDialog(
      context: context,
      barrierDismissible: false, dialog
      builder: (BuildContext context) {
        return Dialog(
          child: Container(
            padding: EdgeInsets.all(20),
            child: Column(
              mainAxisSize: MainAxisSize.min,
              children: [
                Text(label),
                SizedBox(height: 20),
                ElevatedButton(
                  onPressed: () {
                    Navigator.of(context).pop(); // Close the dialog
                  },
                  child: Text('Cancel'),
                ),
              ],
            ),
          ),
        );
      },
    );
  }

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(
        title: Text('Fashion Mnist'),
      ),
      body: Column(
        children: <Widget>[
          Expanded(
            child: Container(
              child: FutureBuilder<void>(
                future: _initializeControllerFuture,
                builder: (context, snapshot) {
                  if (snapshot.connectionState == ConnectionState.done) {
                    return CameraPreview(_controller);
                  } else {
                    return Center(child: CircularProgressIndicator());
                  }
                },
              )
            ),
          ),
          Row(
            mainAxisAlignment: MainAxisAlignment.spaceEvenly,
            children: [
              ElevatedButton.icon(
                onPressed: _takePicture,
                icon: Icon(Icons.camera_alt),
                label: Text('Take Picture'),
              ),
              ElevatedButton.icon(
                onPressed: _pickImage,
                icon: Icon(Icons.photo_library),
                label: Text('Upload from Gallery'),
              ),
            ],
          ),
        ],
      ),
    );
  }
  Future<void> _initTensorFlow() async{
    String? res = await Tflite.loadModel(
        model: "assets/model/model.tflite",
        labels: "assets/model/labels.txt",
        numThreads: 1,
        isAsset: true,
        useGpuDelegate: false
    );
  }
  Future<String> _objectRecongnition(String filepath) async{
    var recognitions = await Tflite.runModelOnImage(
        path: filepath,
        numResults: 10,
        threshold: 0.1,
        asynch: true
    );
    return recognitions![0]["label"].toString();
  }
}

Create and Release a Private Python Package on GitHub

Abdellah Hallou — Sun, 20 Oct 2024 12:31:56 +0000

Introduction

As software engineers, we often find ourselves reusing code across different modules and projects. But let's face it, this repetition creates a challenge: when we need to adjust or fix that code, we have to make the same changes in multiple places. For those of us who value efficiency and automation, the solution is clear - create a separate package that can be installed and used across our projects.
However, when dealing with confidential code, we can't simply publish our package on public repositories like PyPI. Instead, we need to deploy it to a private repository such as GitHub or GitLab. This approach allows us to maintain security while still benefiting from the convenience of a reusable package.

In this tutorial, we’ll guide you through the process of:

Creating a Python package
Deploying the package to a private repository (GitHub)
Installing the package in a virtual environment (venv)

By following these steps, you'll be able to reduce code duplication and simplify maintenance of shared code across your projects.

Note: DRY doesn't just stand for "Don't Repeat Yourself" - it's also a lifestyle choice.

1. Setting Up Your Project Structure

First, let's set up a basic project structure for our Python package:

my-package/
├── my_package/
│   ├── __init__.py
│   └── module1.py
├── setup.py
├── build.pipeline.yml
├── requirements.txt
├── .gitignore
├── README.md
├── MANIFEST.in
└── LICENSE

Let's break down the anatomy of our private Python package. Each file and directory plays a crucial role in making our package functional and installable:

my-package/: This is the root directory of our project. It's like a house that contains all the rooms (files) we need.
my_package/: This subdirectory is where the actual Python code lives. It's named the same as our package for clarity.
- __init__.py: This file makes Python treat the directory as a package. It can be empty or can execute the initialization code for the package.
- module1.py: This is where we put our main code. You can have multiple module files depending on your package's complexity.
setup.py: Think of this as the instruction manual for our package. It contains metadata about our package (like its name and version) and lists its dependencies. This file is essential for making our package installable via pip.
requirements.txt: This file lists all the external Python packages our project depends on. It's like a shopping list for pip, telling it exactly what to install to make our package work.
README.md: This is the welcome mat of our project. It's usually the first thing people see when they visit our GitHub repository, so we use it to explain what our package does, how to install it, and how to use it.
.gitignore: This file tells Git which files or directories to ignore. It's handy for keeping compiled code, temporary files, or sensitive information out of version control.
LICENSE: This file specifies how others can use, modify, or distribute our package. It's crucial for open-source projects and helps protect our work.
MANIFEST.in: This file is used to include non-Python files in our package distribution. If we have data files, documentation, or other resources that need to be included, we list them here.
build.pipeline.yml: This file defines our Continuous Integration/Continuous Deployment (CI/CD) pipeline. It automates tasks like running tests and building the package when we push changes to our GitHub repository.

2. Creating the Package Code

Let's create a simple module within our package. In my_package/module1.py:

class Hello:
    def __init__(self, name):
        self.name = name

    def greet(self):
        return f"Hello, {self.name}!"

In my_package/__init__.py, we'll import our module:

from .module1 import Hello

3. Creating `setup.py`

The setup.py file is crucial for packaging our project. Here's a basic example:

from setuptools import setup, find_packages

with open('requirements.txt') as f:
    requirements = f.read().splitlines()

setup(
    name="my_package",
    version="0.1",
    include_package_data=True,
    python_requires='>=3.8',
    packages=find_packages(),
    setup_requires=['setuptools-git-versioning'],
    install_requires=requirements,
    author="Abdellah HALLOU",
    author_email="abdeallahhallou33@gmail.com",
    description="A short description of your package",
    long_description=open('README.md').read(),
    long_description_content_type="text/markdown",
    classifiers=[
        "Programming Language :: Python :: 3.8",
        "License :: OSI Approved :: MIT License",
        "Operating System :: OS Independent",
    ],
    version_config={
       "dirty_template": "{tag}",
    }
)

4. Creating `requirements.txt`

In our requirements.txt file, we include the necessary dependencies for building and distributing our package:

setuptools==69.2.0
wheel
twine

5. Building and Installing Your Package

Install the requirements. To keep things simple, we will use the Python virtual environment.

python -m venv env
source env/bin/activate # for linux and mac
./env/Scripts/activate # for windows
pip install -r requirements.txt

To build our package:

python setup.py sdist bdist_wheel

To install our package locally for testing:

pip install -e .

You can commit your work and ignore the folders using .gitignore file:

https://github.com/github/gitignore/blob/main/Python.gitignore

6. Publishing the Package on GitHub with a tag

To publish the package, first, create a build.pipeline.yml file at the root of the project my-package/ and commit it. The deployment will be done with twine, the library that we installed before:

name: Python package

on:
  push:
    tags:
      - '*'

jobs:
  build:

    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: ["3.7", "3.8", "3.9", "3.10", "3.11"]
    steps:
      - uses: actions/checkout@v4
      - name: Set up Python 3.8
        uses: actions/setup-python@v5
        with:
          python-version: 3.8
      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          if  [ -f requirements.txt ]; then pip install -r requirements.txt; fi
      - name: Build package
        run: |
          python setup.py sdist bdist_wheel
      - name: Publish package
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          pip install twine
          python -m twine upload --repository-url https://api.github.com/orgs/${{ github.repository_owner }}/packages/pypi/upload dist/*

If you need to include non-Python files with your module installation, you can use a MANIFEST.in file. This file specifies which additional files should be included in your package distribution.

# To include all files and subdirectories within the my_package/directory_name directory
recursive-include my_package/directory_name *
# To include a specific file in your package.
include  my_package/file_name

Then upload the package:

git tag 0.0.1 -m "New Release"
git push origin 0.0.1

7. Install the Package

Create an access token:

Go to Settings > Developer Settings > Personal access tokens (classic) > Generate new token.
Ensure you check the write:packages scope to grant the necessary permissions.

Once you have your token, keep it secure, as you'll need it to install your package.

On your machine, you can install your private package using the following template:

# template
pip install git+https://{{ your access token }}@github.com/{{ username }}/{{ repository name}}.git@{{ tag/version }}#egg={{ package name }}
# example
pip install git+https://{{ your access token }}@github.com/ABDELLAH-Hallou/my-package-repo.git@0.0.1#egg=my_package

Conclusion

Well done, you know now how to create and deploy your own private packages with Python on GitHub.

Github repository link : https://github.com/ABDELLAH-Hallou/Private-Python-Package-Deployment