DEV Community: Meqdad Darwish

ACID Properties in Databases - Part 2: Your Shield Against Transactional Chaos

Meqdad Darwish — Tue, 10 Dec 2024 16:22:06 +0000

Read the first part of this series from here.

The Transactional Concurrency Dilemma

In the dynamic world of database management, concurrency represents the simultaneous execution of multiple transactions across complex systems. While this approach enables high-performance and efficient data processing, it introduces critical challenges that can compromise data integrity. Scenarios like dirty reads, non-repeatable reads, phantom reads, and lost updates threaten the fundamental reliability of database operations, creating potential risks that range from minor inconsistencies to significant financial and operational disruptions.

Understanding these concurrency problems is essential for developers and system architects to design robust, dependable transactional systems.

Dirty Reads

Scenario: The Risky Loan Approval

Sami works as a bank loan officer processing a critical $50,000 loan application. Simultaneously, another system process is attempting to update the applicant's credit status:

Transaction A begins reviewing the loan application
Concurrent Transaction B starts updating the applicant's credit score -Transaction B temporarily marks the credit score as improved (not yet committed)
Transaction A reads this temporary, unconfirmed credit score
Transaction B rolls back the credit score change due to an error

Potential Impact:

Loan might be approved based on a false credit improvement
Critical financial decision made on non-verified information
Significant financial risk for the bank

Non-Repeatable Reads

Scenario: Stock Trading Platform

A day trader monitors a volatile stock in real-time:

First query reveals stock price at $100
Concurrent trading activity triggers price updates
Subsequent read shows stock price at $95
Trader's decision-making becomes unreliable due to inconsistent data views

Potential Impact:

Unpredictable trading strategies
Potential financial losses
Undermined trust in trading platform

Phantom Reads

Scenario: E-commerce Inventory Management

An online retailer processes multiple simultaneous orders for limited stock items:

Initial inventory check shows 5 gaming consoles available Concurrent transactions:
- Transaction A starts inventory verification
- Transaction B adds 2 new gaming consoles to inventory
Transaction A's subsequent read now shows 7 consoles Unexpected inventory changes create reconciliation challenges

Potential Impact:

Overselling potential
Inventory tracking inconsistencies
Customer experience disruption

Lost Update

Scenario: Travel Booking System

Two customers simultaneously attempt to book the last seat on a flight:

Initial seat availability: 1 seat remaining
Customer A begins booking process
Customer B starts identical booking process
System fails to properly manage concurrent writes
One booking transaction silently overwrites the other

Potential Impact:

Potential revenue loss
Customer trust erosion
Operational chaos

Concurrency is not just a technical challenge—it's a critical business risk management issue that demands sophisticated, thoughtful solutions.

Isolation Levels

In the complex landscape of database management, isolation levels represent sophisticated strategies for managing concurrent transactions. These levels define how different transactions interact and view data simultaneously, ranging from minimal protection (Read Uncommitted) to absolute data integrity (Serializable). Each isolation level offers a unique balance between performance and consistency, addressing potential concurrency challenges like dirty reads, non-repeatable reads, and phantom reads.

By understanding and strategically implementing these isolation mechanisms, developers can design robust, reliable transactional systems that maintain data integrity while optimizing system performance.

Read Uncommitted

Scenario: Cryptocurrency Trading Platform

High-frequency trading system processing multiple transactions
No protection against dirty reads
Potential for processing unconfirmed market data

Pros:

Highest performance
Minimal system overhead
Maximum concurrency

Cons:

Highest risk of data inconsistency
Unreliable for critical systems

Use Cases:

Non-critical analytics
High-speed data streaming
Preliminary market tracking

Examples on Databases that can apply it:

SQL Server (configurable)

Read Committed

Scenario: Banking Transaction Processing

Customer checks account balance
Prevents reading uncommitted transactions
Ensures only confirmed financial data is visible

Pros:

Prevents dirty reads
Default isolation in many databases
Reasonable performance-consistency balance

Cons:

Allows non-repeatable reads
Potential for phantom read issues

Use Cases:

Standard financial applications
E-commerce platforms
General transactional systems

Examples on Databases that can apply it:

PostgreSQL (default)
Oracle, MySQL

Repeatable Read (Sometimes called: Snapshot Isolation)

Scenario: Inventory Management System

Warehouse manager generating stock reports
Guarantees consistent data view during entire transaction
Prevents other transactions from modifying viewed data

Pros:

Consistent data snapshots
Prevents non-repeatable reads
Suitable for analytical processes

Cons:

Potential performance overhead
Risk of deadlocks
Allows phantom reads

Use Cases:

Detailed financial reporting
Batch processing
Complex data analysis

Examples on Databases that can apply it:

MySQL (InnoDB Engine)
PostgreSQL

Serializable

Scenario: Global Financial Compliance Reporting

Generating regulatory financial statements
Complete isolation between concurrent transactions
Prevents all concurrency anomalies

Pros:

Absolute data consistency
Eliminates all read phenomena
Guaranteed transaction integrity

Cons:

Significant performance impact
High resource consumption
Potential system bottlenecks

Use Cases:

Critical financial systems
Regulatory compliance reporting
High-stakes transactional environments

Examples on Databases that can apply it:

SQL Server
Oracle

Before deciding which isolation level to use, CONSIDER...

Isolation level selection involves balancing:

Performance requirements
Data consistency needs
Specific application constraints
Potential concurrency challenges

Developers must carefully evaluate their system's unique requirements, understanding that higher isolation levels provide more protection at the cost of reduced performance.

More about Durability

Database durability represents a critical mechanism ensuring that committed transactions survive system failures, power outages, and unexpected interruptions. These techniques create robust data protection strategies that guarantee information integrity, allowing databases to recover and maintain consistent state even under extreme operational conditions.

Here are some of popular Durability techniques in database systems:

Write-Ahead Logging (WAL)

Concept: Logs all changes before applying them to the main database
Purpose: Enables complete transaction recovery
Databases: PostgreSQL, SQLite
Mechanism:

Records transaction details in sequential log
Allows replay of operations after system restart
Ensures no committed transaction is lost

Transaction Logs

Concept: Maintains sequential record of database modifications
Purpose: Supports point-in-time recovery
Databases: SQL Server, Oracle
Mechanism:

Captures every database state change
Provides rollback and recovery capabilities
Allows restoration to specific historical moment

There are more durability techniques, I'll put the extra resources at the bottom of article.

Data Consistency vs. Reading Consistency

Data Consistency

Data consistency ensures that information remains accurate, complete, and reliable across different system components and operations. It prevents data corruption and maintains system trustworthiness by ensuring that all copies of data remain synchronized and valid.

Maintaining Data Consistency:

Implement atomic transactions that complete entirely or not at all
Use strict validation mechanisms before data modifications
Apply locking strategies to prevent concurrent conflicting updates
Utilize versioning to track and reconcile data changes
Implement rollback mechanisms for failed or partial updates

Reading Data in Distributed Systems

In distributed systems, reading data becomes complex when multiple data sources or nodes can serve the same information. This creates a fundamental challenge: how to ensure data read consistency while maintaining system performance and availability?

Eventual Consistency

Eventual consistency emerges as a pragmatic solution, accepting temporary data divergence with a guarantee of final synchronization. It prioritizes system responsiveness and availability over immediate, strict consistency.

Example: Social Media Post Synchronization

User posts content in Qabilah (An Arabic social network)
Servers in different global regions temporarily show slightly different data
Within milliseconds to seconds, all servers converge to the same consistent state
System prioritizes quick user experience over immediate perfect synchronization

By accepting short-term inconsistencies, eventual consistency allows distributed systems to remain responsive, scalable, and resilient, making it crucial for high-performance, globally distributed applications.

Generally;

Database transactions represent a critical intersection of performance and data integrity. The challenges of concurrent systems demand sophisticated strategies to protect information reliability.

Main Insights

Concurrency Risks
- Dirty reads
- Non-repeatable reads
- Phantom reads
- Lost updates
Strategic Considerations
Successful transaction management requires balancing performance, consistency, and system-specific constraints. Isolation levels are not one-size-fits-all solutions but carefully calibrated mechanisms.

Final Word

Transactional systems are more than technical mechanisms—they're essential risk management tools. Success depends on understanding potential failure modes and implementing intelligent mitigation strategies.

The complexity of database transactions demands continuous learning and adaptive approaches. By recognizing the delicate balance between system performance and data integrity, developers can build robust, reliable computing environments.

Extra Resources

ACID Properties in Databases: What Happens Without Them?

Meqdad Darwish — Wed, 27 Nov 2024 12:41:29 +0000

Intro...

ACID is a set of principles that ensure databases operate reliably, especially when handling critical data. These principles stand for:

Atomicity: Ensures transactions are "all or nothing."
Consistency: Maintains the validity of data by following defined rules.
Isolation: Prevents interference between concurrent transactions.
Durability: Guarantees committed transactions remain safe, even during failures.

To understand ACID properties, it’s helpful to grasp these foundational concepts:

1. What is a Database Transaction?

A transaction is a sequence of operations performed on a database, treated as a single logical unit (unit of work). For example:

Transferring money between bank accounts involves deducting from one account and adding to another.

Here’s a quick SQL example of transferring $500 from one account to another:

BEGIN;  -- Start the transaction

-- Deduct $500 from the checking account
UPDATE accounts
SET balance = balance - 500
WHERE account_id = 1;

-- Add $500 to the savings account
UPDATE accounts
SET balance = balance + 500
WHERE account_id = 2;

COMMIT;  -- Commit the transaction

If something goes wrong (e.g., insufficient funds in the checking account), you can roll back the transaction:

ROLLBACK;  -- Undo all changes made in the transaction

A transaction must either fully succeed (commit) or fail entirely (rollback) to prevent partial changes.

2. Why Are Transactions Important?

Transactions ensure data reliability, especially in multi-user or high-load systems. Without them, systems can lose, corrupt, or display inconsistent data.

3. Concurrency in Databases

When multiple users or processes interact with the database at the same time, it’s called concurrency. Managing concurrent access is critical to maintaining data integrity, which is where ACID plays a vital role.

1. Atomicity: What if Transactions Break in the Middle?

Scenario:

You’re transferring $1,000 from your Savings account to your Checking account. Here’s what happens:

The database deducts $1,000 from Savings.
Before it adds $1,000 to Checking, the system crashes.

Without Atomicity, your money disappears from Savings, but it’s not reflected in Checking. This partial transaction leaves your data in an inconsistent and unacceptable state.

The Solution – Atomicity

Atomicity ensures that transactions are "all or nothing." If any part of the transaction fails, the database rolls back all changes, leaving the data untouched.

How Databases Handle It:

PostgreSQL and MySQL (InnoDB) use Write-Ahead Logging (WAL) to track and roll back incomplete transactions, ensuring full ACID compliance across entire transactions.
Oracle uses a combination of undo segments and redo logs to manage atomicity, supporting complex transaction scenarios including distributed transactions.
NoSQL systems like MongoDB initially supported atomicity only at the document level, but now offer multi-document transactions with ACID guarantees in recent versions (4.0+), though with some limitations compared to traditional relational databases.

2. Consistency: What if Rules Are Violated?

Scenario:

Imagine an e-commerce database that has a rule: Total Inventory = Warehouse A + Warehouse B.

A transaction updates Warehouse A’s count but fails to update Warehouse B’s count due to an error.
The Total Inventory column no longer reflects the actual stock, leading to incorrect reports and potential overselling.

Without Consistency, data integrity is compromised, and your system’s credibility suffers.

The Solution – Consistency

Consistency ensures every transaction moves the database from one valid state to another, always adhering to defined rules, constraints, and relationships.

How Databases Handle It:

PostgreSQL enforces consistency through comprehensive constraint mechanisms, including primary keys, foreign keys, check constraints, and complex trigger-based validations.
MySQL (InnoDB) maintains consistency using table-level and column-level constraints, supporting referential integrity and structured validation rules.
MongoDB's consistency model is distinctively flexible and places significant responsibility on the application developer. Unlike traditional relational databases with rigid built-in constraints, MongoDB relies heavily on application-level validation for maintaining data integrity. While MongoDB provides JSON Schema validation as a mechanism for defining structural and content rules at the database level, many consistency checks and complex relational validations must be implemented by developers in their application logic.

3. Isolation: What if Transactions Clash?

Scenario:

Two bank tellers are processing withdrawals from the same account:

Teller A withdraws $500 from an account with $1,000.
Teller B simultaneously withdraws $600.

Without Isolation, both transactions might read the initial balance of $1,000 and allow the withdrawals, leading to an overdrawn account, even if the bank doesn’t permit overdrafts.

The Solution – Isolation

Isolation ensures that concurrent transactions don't interfere with each other, maintaining data consistency. Transactions are executed as if they happen one after the other, even if processed concurrently.

How Databases Handle It:

PostgreSQL leverages Multi-Version Concurrency Control (MVCC) to manage concurrent transactions. This approach creates snapshot versions of data, allowing multiple transactions to read and write without blocking each other. MVCC ensures that transactions see a consistent view of the database at the moment they begin, preventing interference between concurrent operations.
MySQL (InnoDB) implements isolation through a combination of row-level locking and MVCC techniques. This approach allows multiple transactions to access different rows simultaneously, minimizing contention. The default Repeatable Read isolation level ensures that transactions can read consistent data without being affected by concurrent modifications.
MongoDB initially provided isolation at the document level, meaning operations on a single document were atomic. With the introduction of multi-document ACID transactions in version 4.0, MongoDB expanded its isolation capabilities. However, isolation in MongoDB still relies significantly on application-level design, with developers needing to carefully manage transaction boundaries and potential conflicts.

4. Durability: What if Data Disappears After a Crash?

Scenario:

You complete a purchase online and receive a confirmation email. Moments later, the server crashes. When the system restarts, your purchase data is gone, and you’re left wondering if your order went through.

Without Durability, committed transactions can be lost, undermining user trust and causing operational chaos.

The Solution – Durability

Durability ensures that once a transaction is committed, it's permanently saved, even in the event of a crash or power failure.

How Databases Handle It:

PostgreSQL uses Write-Ahead Logging (WAL) to ensure all changes are written to the transaction log before being acknowledged. This means every database modification is first recorded in a sequential log file, which can be used to reconstruct data in case of system failure, guaranteeing that committed transactions are not lost.
MySQL (InnoDB) uses redo logs and checkpoints to persist transactions. The redo log records all changes made to the database, while checkpoints periodically synchronize these changes to the actual data files. This approach ensures that even if the server crashes, committed transactions can be recovered by replaying the redo log.
MongoDB uses journaling to ensure durability at the document level, with configurable write concerns that allow developers to balance between durability and performance. The journal maintains a log of write operations, enabling the database to recover and replay transactions in the event of an unexpected shutdown, though the default settings can be tuned based on specific application requirements.

Why ACID Matters for Developers?

Practical Importance

ACID properties are critical safeguards that protect your data and ensure system reliability. They're not just theoretical concepts, but practical tools for maintaining data integrity across complex applications.

Key Developer Insights

Understand how your chosen database implements ACID principles
Recognize the trade-offs between data consistency and performance
Design transactions with reliability as a primary concern

Critical Application Scenarios

ACID is crucial in:

Financial systems
E-commerce platforms
Healthcare record management
Inventory tracking
Critical multi-step data operations

Best Practices

Always use transactions for complex data updates
Implement robust error handling
Choose database technologies that match your reliability needs
Test transaction behavior under various failure scenarios

When to Be Flexible

Some modern systems use alternative models like BASE (Basically Available, Soft state, Eventual consistency) for improved scalability, especially in high-traffic or distributed environments.

Final Word...

Understanding ACID isn't about memorizing acronyms—it's about developing a robust approach to data management that prevents costly errors and maintains system integrity.

Extra Resources

Context Managers Demystified: Simplify Your Resource Handling in Python

Meqdad Darwish — Fri, 15 Nov 2024 18:23:10 +0000

Intro...

Context managers in Python are a powerful tool that allow you to manage the setup and teardown of resources in a safe and efficient manner. They provide a way to ensure that resources are properly initialized, used, and cleaned up, even in the face of exceptions or unexpected control flow.

The main benefit of using context managers is that they help you write more robust, maintainable, and Pythonic code. By encapsulating resource management logic, context managers make it easier to ensure that resources are properly handled, reducing the risk of resource leaks or inconsistent state.

Common Use Cases

Some common use cases for context managers include:

File handling (e.g., open())
Database connections
Locking mechanisms
Temporary directory management
Profiling and timing code execution

The `with` Statement

The primary way to use a context manager in Python is with the with statement. The with statement provides a convenient syntax for working with context managers, allowing you to focus on the core logic of your code rather than worry about resource management.

Here's an example of using the with statement to open a file:

with open('data.txt', 'r') as file:
    content = file.read()
    print(content)

When the with block is exited, either normally or due to an exception, the context manager's __exit__() method is automatically called, ensuring that the file is properly closed.

Implementing a Custom Context Manager

To create a custom context manager, you need to define a class with two special methods: __enter__() and __exit__(). The __enter__() method is responsible for setting up the resource, while the __exit__() method is responsible for cleaning up the resource.

Here's an example of a custom context manager that manages a PostgreSQL database connection:

import psycopg2

class PostgresManager:
    def __init__(self, host, port, database, user, password):
        self.host = host
        self.port = port
        self.database = database
        self.user = user
        self.password = password
        self.conn = None

    def __enter__(self):
        self.conn = psycopg2.connect(
            host=self.host,
            port=self.port,
            database=self.database,
            user=self.user,
            password=self.password
        )
        return self.conn

    def __exit__(self, exc_type, exc_value, traceback):
        if self.conn:
            if exc_type is None:
                self.conn.commit()
            else:
                self.conn.rollback()
            self.conn.close()
        return False

In this example, the __enter__() method establishes a connection to the PostgreSQL database using the provided connection parameters, and the __exit__() method is responsible for committing or rolling back the transaction, depending on whether an exception occurred, and then closing the connection.

The __exit__() method receives three arguments: exc_type, exc_value, and traceback, which provide information about any exception that occurred within the with block. In this example, if an exception occurs, the method rolls back the transaction; otherwise, it commits the transaction.

By using this custom context manager, you can simplify database interactions and ensure that connections are properly closed, even in the face of exceptions:

with PostgresManager('localhost', 5432, 'demo_db', 'myuser', 'mypassword') as conn:
    with conn.cursor() as cursor:
        cursor.execute("CREATE TABLE customers (id SERIAL PRIMARY KEY, name TEXT)")
        cursor.execute("INSERT INTO customers (name) VALUES (%s)", ('Meqdad',))
        # Other database operations here...

This approach helps you write more robust and maintainable code by encapsulating the database connection management logic within the context manager.

Context Manager Generators

In addition to creating custom context manager classes, you can also create context managers using generator functions. This approach can be more concise and easier to read in some cases.

Here's an example of a context manager generator that manages a lock:

from contextlib import contextmanager
from threading import Lock

@contextmanager
def lock_manager(lock):
    lock.acquire()
    try:
        yield lock
    finally:
        lock.release()

In this example, the @contextmanager decorator is used to define a generator function that acts as a context manager. The yield statement is used to define the point at which control is transferred to the with block, and the finally block ensures that the lock is released, even if an exception occurs.

Context Management in the Standard Library

The Python standard library provides a number of built-in context managers that you can use in your code. These include:

open(): Manages the opening and closing of files.
lock(): Manages the acquisition and release of locks.
ThreadPoolExecutor(): Manages the creation and cleanup of a pool of worker threads.

Using these built-in context managers can help you write more concise and reliable code, as the resource management logic is already implemented for you.

Best Practices and Considerations

When working with context managers, there are a few best practices and considerations to keep in mind:

Error Handling and Cleanup: Ensure that your __exit__() method properly handles exceptions and cleans up resources, even in the face of unexpected errors.
Nesting Context Managers: You can nest context managers within each other, which can be useful when managing multiple resources.
Performance Considerations: While context managers are generally efficient, be mindful of any overhead introduced by the setup and teardown of resources, especially in performance-critical code.

Final Word

Context managers are a powerful tool in the Python ecosystem, allowing you to write more robust, maintainable, and Pythonic code.

By encapsulating resource management logic, context managers help you ensure that resources are properly handled, reducing the risk of resource leaks or inconsistent state.

Whether you're working with built-in context managers or creating your own custom ones, understanding the fundamentals of context managers will help you write cleaner, more efficient, and more reliable Python code.

Stop Creating Repos To Share Your Code Snippets - Use GitHub Gists Instead

Meqdad Darwish — Fri, 08 Nov 2024 20:02:32 +0000

You know the feeling - you've got this super useful code snippet that you want to share with your team, but the thought of setting up a whole new GitHub repository feels like overkill. Well, I've got the perfect solution for you: GitHub Gists.

What Are GitHub Gists?

Gists are like mini, lightweight repositories designed specifically for sharing small bits of code, text, or other content. Instead of going through all the hassle of initializing a new repo, adding a README, setting up branches, and all that other stuff, with Gists you can just paste your code and go.

Why Use Gists Instead of Repos?

There are a few key reasons why Gists are way better than creating a new repo every time you want to share something:

Super quick to set up: No repo initialization, no complex folder structures, just paste your code and hit "Create Gist".
Easy to find and organize: All your Gists are in one place, searchable, and you can even add descriptions to keep things tidy.
Version control built-in: Gists use Git, so you get commit history, forking, and all the version control goodness without the overhead.
Minimal maintenance: No need to worry about managing a full repo - Gists are self-contained and easy to update.

What Can You Do With Gists?

The possibilities are endless! Here are just a few ways developers use Gists:

🔍 Share Code Snippets: Whether it's a handy function, a code example, or a solution to a problem, Gists make it easy to quickly paste and share your code.
🛠️ Store Configuration Files: Keep your .bashrc, .vimrc, or other config files in a Gist so you can access them from anywhere.
📚 Create Living Documentation: Write up tutorials, explanations, or personal notes in Markdown, and embed code snippets right in the Gist.
🐞 Collaborative Debugging: When you're trying to track down a tricky bug, share the problematic code in a Gist and let your teammates fork it and help you out.

How to Get Started with Gists

Using Gists is a breeze. Here's a quick step-by-step:

Go to gist.github.com
Paste your code, text, or other content into the box
Add a brief description to help you (and others) remember what it's for
Choose whether you want it to be public or "secret" (i.e. only accessible via the link)
Click "Create Gist" and you're done!

You can even add multiple files to a single Gist if you've got related snippets or bits of code that work better together.

Gist Features You Should Know

Gists are packed with handy features that make them even more powerful:

Embedding: You can easily embed Gists into websites or documents
Version Control: Every change creates a new version that you can refer back to
Forking: Others can make their own copy of your Gist to modify and improve
Commenting: Collaborators can leave feedback and suggestions right on the Gist
Starring: Save Gists you want to reference later by starring them

When to Use Gists (and When Not To)

✅ Use Gists for:

Sharing solutions to common problems
Storing useful code snippets
Quick collaboration on small pieces of code
Saving configuration files
Creating simple tutorials

❌ Don't use Gists for:

Full-fledged projects
Code that needs extensive collaboration
Anything requiring advanced issue tracking
Private/sensitive code (remember, "secret" ≠ "private")

Wrap Up

Gists are like the secret superpower in every developer's toolbox. They make it a breeze to share, save, and collaborate on small bits of code, without all the overhead of a full GitHub repository.

The next time you're tempted to create a new repo just to share a code snippet, stop and ask yourself, "Wouldn't a Gist be better for this?" 99% of the time, the answer is a resounding YES!

So go forth and start using Gists!

Lessons I’ve Learned: 10 Tips for Successful Open-Source Project Development

Meqdad Darwish — Tue, 29 Oct 2024 19:38:21 +0000

Open-source software development can be an incredibly rewarding experience, allowing you to share your work with the world and contribute to the broader technical community. However, there are a few important lessons I’ve learned along the way that I’d like to share.

Whether you’re working on your first open-source project or looking to improve your existing one, these tips can help set you up for success.

Build and Publish, Don’t Be a Perfectionist: It’s easy to get caught up in trying to make your project perfect before releasing it. However, the best way to get feedback and improve your work is to put it out there for others to use. Don’t wait until it’s “ready” — publish early and often, gathering user feedback to guide your development.
Find a Problem and Solve It, Even If It’s Small: The most successful open-source projects tend to solve real, tangible problems that users are facing. Don’t feel like you need to build the next groundbreaking framework — even small, focused solutions can be incredibly valuable. Identify a need, build something that addresses it, and you’re on your way.
Choose the Right Name: The name of your project can have a big impact on its discoverability and adoption. Think about how users will search for solutions to the problem you’re solving, and pick a name that will help them find your work. Avoid overly generic or obscure names in favor of something clear and descriptive.
Learn Versioning for Your Package Releases: Proper versioning is crucial for open-source projects, as it allows users to understand the changes between releases and decide when to upgrade. Follow semantic versioning (major.minor.patch) and be diligent about incrementing the version number with each release.
Write Release Notes: Along with versioning, providing clear and detailed release notes helps users understand what’s changed in each update. Include information about new features, bug fixes, and any breaking changes. This transparency builds trust and makes it easier for users to adopt your project.
Publish on Platforms that Help Users Discover and Use Your Work: Don’t just upload your project to GitHub and call it a day. Look into the package managers and distribution platforms specific to your language or ecosystem (e.g., PyPI for Python, npm for Node.js). These make it easy for users to find, install, and use your project.
Maintain Backward Compatibility: While it’s important to continually improve and evolve your project, you also need to consider the users who are relying on your existing code. Whenever possible, ensure that new versions maintain backward compatibility, making it easy for existing users to upgrade without breaking their legacy applications.
Build Comprehensive Documentation: Thorough documentation is essential for the long-term success of your open-source project. Provide clear and concise instructions on how to install, configure, and use your project. Include examples, tutorials, and explanations of key features and functionality. Well-written documentation makes it easier for users to understand and adopt your work and also helps onboard new contributors to your project.
Establish Contribution Guidelines: As your project grows, contributors can be a valuable asset. Setting clear contribution guidelines helps other developers know how to add to your project. A “CONTRIBUTING.md” file explaining coding standards, pull request guidelines, and testing expectations will streamline collaboration and keep contributions consistent with your vision.
Create a Future Plan for Your Project: Think about the project’s roadmap and what features or improvements could be useful in the future. Brainstorm ideas as they come to you and outline these in a “roadmap” section of your README or documentation. This shows users (and potential contributors) your vision and goals, making it easier for them to understand where they might add value.

By keeping these lessons in mind, you can set your open-source project up for long-term success, building a community of users and contributors who appreciate your hard work and dedication.

Mastering MongoDB Aggregation: A Journey from Fundamentals to Advanced Optimization

Meqdad Darwish — Sun, 20 Oct 2024 11:55:38 +0000

MongoDB’s Aggregation Framework is one of the most powerful features for data processing and analysis in modern databases. In this comprehensive guide, we’ll explore what it is, how to use it effectively, and advanced optimization techniques that can significantly improve your application’s performance.

=================

Understanding the Basics
Building Your First Pipeline
Advanced Optimization Techniques

Understanding the Basics

========================

What is MongoDB Aggregation Framework?

The MongoDB Aggregation Framework is a powerful tool designed for handling complex data analysis tasks directly within the database. It allows you to process and transform large volumes of data by passing documents through a series of stages, where each stage applies a specific operation. You can think of it like an assembly line in a factory: each stage refines or modifies the data before passing it on to the next, enabling you to perform tasks like filtering, grouping, and calculating results — all in one streamlined flow.

Why Use Aggregation Framework?

Traditional CRUD operations can fall short when you’re dealing with more complex data processing tasks. That’s where MongoDB’s Aggregation Framework comes in, offering several key advantages:

Data Transformation: Perform complex calculations and reshape your data as needed.
Performance Optimization: Handle heavy processing directly at the database level, minimizing overhead.
Flexibility: Chain multiple operations together in a single, efficient query.
Scalability: Easily manage and process large datasets with high efficiency.

Common Use Cases

Real-Time Analytics Dashboards
Report Generation
Data Warehousing and ETL Operations
Complex Data Transformations
Statistical Analysis

Building Your First Pipeline

Let’s walk through a practical example using an e-commerce dataset. We’ll build a pipeline that analyzes sales performance across different product categories.

Sample Data Structure

{
  _id: ObjectId("..."),
  product: "Gaming Laptop",
  category: "Electronics",
  price: 1299.99,
  sales: {
    quantity: 5,
    date: ISODate("2024-10-17"),
    location: "Al-Quds"
  },
  specs: {
    brand: "TechPro",
    model: "GX-500",
    features: ["16GB RAM", "1TB SSD", "RTX 4060"]
  }
}

Building the Pipeline Step by Step

1. Filtering Relevant Data

Before diving into any transformations, the first step is to reduce the dataset by filtering only the data that meets certain criteria. This is typically done using the $match stage, which works like a query filter. For example, let’s filter for recent sales after January 1, 2024:

db.products.aggregate([  {
    $match: {
      "sales.date": {
        $gte: ISODate("2024-01-01")
      }
    }
  }
])

This $match stage ensures that only the documents containing sales data from 2024 onward are passed to the next step. By doing this, we reduce the amount of data that subsequent stages will need to process, which improves performance. It’s especially important in large datasets to filter early on to avoid unnecessary computations.

2. Reshaping the Data

Once the relevant data has been filtered, the next step is to restructure the documents to focus only on the fields that are necessary for analysis or reporting. This is done using the $project stage. Here, we’ll extract the product, category, and calculate the total revenue for each sale, as well as include the sales location:

db.products.aggregate([  {
    $match: {
      "sales.date": {
        $gte: ISODate("2024-01-01")
      }
    }
  },
  {
    $project: {
      product: 1,  // Include the product name or ID
      category: 1,  // Include the product category
      totalRevenue: {
        $multiply: ["$price", "$sales.quantity"]
      },  // Calculate total revenue by multiplying price and quantity sold
      location: "$sales.location"  // Include the location of the sale
    }
  }
])

In this step, we use $project to reshape each document. By calculating totalRevenue directly in the aggregation, we avoid the need to perform this calculation outside the database, saving time and resources. This stage is crucial when you want to work with transformed data or computed fields for analysis purposes.

3. Grouping and Analysis

Now that we have the filtered and reshaped data, the next logical step is to aggregate or summarize it. For example, you might want to calculate the total revenue for each product across all locations. This is where the $group stage comes into play:

db.products.aggregate([  {
    $match: {
      "sales.date": {
        $gte: ISODate("2024-01-01")
      }
    }
  },
  {
    $project: {
      product: 1,
      category: 1,
      totalRevenue: {
        $multiply: ["$price", "$sales.quantity"]
      },
      location: "$sales.location"
    }
  },
  {
    $group: {
      _id: "$product",  // Group by product
      totalSales: { $sum: "$totalRevenue" },  // Sum the total revenue for each product
      salesCount: { $sum: 1 },  // Count the number of sales per product
      uniqueLocations: { $addToSet: "$location" }  // Collect unique sales locations
    }
  }
])

In this $group stage:

We group the data by product, indicated by _id: "$product".
The totalSales field accumulates the total revenue per product using $sum.
We also calculate salesCount by summing 1 for each sale, giving us a count of the sales transactions.
Additionally, we use $addToSet to gather all unique locations where the product was sold.

This step is useful for summarizing your data to gain insights like total sales per product, the number of transactions, or even understanding in which locations the product is performing well.

4. Final Formatting

In this final step, we’ll wrap up the aggregation pipeline by applying sorting and fine-tuning the output format to make it more presentable. This involves sorting the results by average revenue in descending order and then adjusting the fields for the final output.

db.products.aggregate([  {
    $match: {
      "sales.date": {
        $gte: ISODate("2024-01-01")
      }
    }
  },
  {
    $project: {
      product: 1,
      category: 1,
      totalRevenue: {
        $multiply: ["$price", "$sales.quantity"]
      },
      location: "$sales.location"
    }
  },
  {
    $group: {
      _id: "$category",  // Group by category instead of product
      totalSales: { $sum: "$sales.quantity" },  // Sum the total quantity sold in each category
      averageRevenue: { $avg: "$totalRevenue" },  // Calculate average revenue per category
      locations: { $addToSet: "$location" }  // Collect unique sales locations per category
    }
  },
  {
    $sort: { averageRevenue: -1 }  // Sort by average revenue in descending order
  },
  {
    $project: {
      category: "$_id",  // Rename _id to category for readability
      totalSales: 1,  // Include the total sales count
      averageRevenue: { $round: ["$averageRevenue", 2] },  // Round average revenue to 2 decimal places
      locations: 1,  // Include the list of locations
      _id: 0  // Exclude the _id field from the final output
    }
  }
])

Explanation:

Sorting the Results: We use the $sort stage to order the results based on averageRevenue in descending order (-1). This ensures that categories with the highest average revenue appear at the top of the list.
Formatting the Output: The final $project stage is used to format the output:
- We rename the _id field to category for better clarity.
- We include totalSales, averageRevenue, and locations in the final output.
- The averageRevenue is rounded to two decimal places using $round.
- The _id field is excluded from the final output by setting it to 0.

This step is essential for making the results more readable and useful for reporting or further analysis. Sorting and rounding help present the data in a clean, organized manner, perfect for dashboards or exporting to reports.

Advanced Optimization Techniques

1. Pipeline Order Optimization

The order of operations in your aggregation pipeline can significantly impact performance. By optimizing this order, you can reduce the amount of data processed in subsequent stages, enhancing efficiency.

// Inefficient Pipeline
[  { $project: { /* large document transformation */ } },
  { $match: { /* could filter out 90% of documents */ } }
]
// Optimized Pipeline
[  { $match: { /* filter first */ } },
  { $project: { /* transform fewer documents */ } }
]

By placing the $match stage first, we filter the dataset to include only relevant documents before transformation.

Why It Matters?

Filtering early reduces the dataset size for subsequent operations. For example, if your dataset has a million documents and your $match condition can exclude 90%, only 100,000 documents will undergo the heavier transformations in the $project stage. This change can lead to significant performance improvements, especially in large datasets.

2. Index Utilization

When it comes to boosting the performance of your aggregation operations in MongoDB, proper indexing is key. Think of indexes as helpful signposts that guide the database to the data it needs without sifting through everything.

For instance, you can create indexes on fields you frequently query, like this:

// Create indexes for frequently used fields
db.products.createIndex({ "sales.date": 1 })
db.products.createIndex({ category: 1, "sales.date": 1 })

Here, we’re setting up an index for sales.date and a compound index for category along with sales.date. This tells MongoDB, "Hey, these fields are important, so look here first!"

Why It Matters

Having the right indexes means MongoDB can quickly find the relevant documents instead of doing a slow, resource-heavy scan of the entire collection. This is especially crucial if you’re working with large datasets. Without proper indexing, your queries can slow down dramatically, leading to frustrating wait times. So, take the time to set up your indexes right.

3. Memory Management Strategies

When dealing with large datasets in MongoDB, effective memory management is essential to keep your queries running smoothly. Here are a couple of strategies that can help:

Use Limits with Sorts:

{ $sort: { score: -1 } },
{ $limit: 10 }

This snippet sorts documents by score in descending order and then limits the results to just the top 10. By doing this, you minimize the amount of data processed, which can significantly reduce memory usage.

However, if you need to ensure that the sorted results are accurate before applying the limit (like when you're retrieving the top 10 items), sorting first is necessary. Keep in mind that sorting large datasets can be costly in terms of performance. Sorting before limiting will ensure you get the correct results, but it will process more data, which might affect query speed.

2. Batch Processing:

Batch processing helps you manage large volumes of data by grouping it into manageable chunks. For instance:

{ $bucket: {
    groupBy: "$price",
    boundaries: [0, 100, 500, 1000],
    default: "Other"
  }
}

In this example, the $bucket stage groups documents based on their price. Instead of processing each document individually, you handle them in batches. This approach reduces memory overhead and keeps your operations efficient.

By using these memory management techniques, you can optimize how your queries interact with large datasets. Limiting the data processed and grouping it into batches makes your MongoDB operations more efficient, helping to prevent memory issues and enhance overall performance.

4. Advanced Optimization Tips

Use Covered Queries:

Ensure all required fields are in the index
Exclude unnecessary fields early in the pipeline

Leverage Compound Indexes:

Create indexes that support multiple pipeline stages
Order index fields based on query patterns

Monitor and Analyze:

Use explain() to understand pipeline execution
Monitor memory usage and response times

Conclusion

The MongoDB Aggregation Framework is a powerful tool that can transform how you process and analyze data. By understanding its capabilities and following optimization best practices, you can build efficient and scalable data processing pipelines.

Remember:

Start with the end goal in mind
Filter early, project wisely
Monitor and optimize performance
Use indexes strategically
Test with representative data volumes

If you're looking for more MongoDB tips and resources, check out my GitHub repository, which includes a handy cheat sheet for popular commands and open-source projects.

Thank you for reading! If you found this article helpful, be sure to follow me on LinkedIn / X / Medium / Dev.to for more valuable insights and technical content. Let’s connect and continue learning together

Building Simple and Customizable Image Classifier with Teachable Machine and Python

Meqdad Darwish — Sat, 02 Sep 2023 16:33:45 +0000

Opening...

Image classification plays a significant role within the field of computer vision, contributing to the resolution of numerous real-world problems. Its applications span various fields and disciplines, as we will explore in the following sections.

BUT...

What is Image Classification?

Image classification involves categorizing images -an entire image- based on their types or classes, and it serves as a fundamental task in the field of computer vision. This process finds wide-ranging applications in various domains, including object detection, medical diagnosis, security, agriculture, and entertainment.

Required Skills/Concepts Before Reading

Before working on this tutorial, you need to know the following skills and concepts:

Python Programming
Image Processing (Basics of OpenCV)
Familiarity with Machine Learning Lifecycle

Note: This code is tested on Windows environment.

Steps for Building Image Classifier

(In our case)

To build an image classifier, you need to walkthrough the following steps:

Collecting the data (images for each class)
Training your model (Teachable Machine from Google)
Testing your results in browser
Exporting trained model into Python environment
Classifying images with trained model by using OpenCV and Teachable Machine packages
Improving and repeat if needed

What We Are Going To Build?

We will create an image classifier that can distinguish between Arabic and English books.

We will create an machine learning model that can classify Arabic and English books. To collect, train, and test data, we will use Teachable Machine from Google.

After training the model, we will export it with a .h5 extension and use it with OpenCV and Teachable Machine Python packages.

What is Teachable Machine?

From the official website:

Teachable Machine is a web-based tool that makes creating machine learning models fast, easy, and accessible to everyone.

With Teachable Machine, you can classify images, sounds, and poses. Our focus will be on using the image classification tool within the platform.

Using Teachable Machine, we will tackle the first three steps: collect data, train the model, and evaluate the results directly in the browser.

Starting Image Classification on Teachable Machine

After opening Teachable Machine platform, select Image Project.

Teachable Machine offers two options to cater to different environments where image classification applications can be deployed. These environments include embedded systems with limited computing power and computers with high computing power.

We will select the Standard image model option for our choice.

Collecting Data (Images)

As shown in the image below, on the left side we can add the classes and their data samples. You can provide samples using camera or from stored images on your computer. We will upload our image samples using camera for each class.

We utilized the built-in computer camera to capture screenshots of various books in both Arabic and English, as depicted in the accompanying images.

Arabic books:

The first one:

The second one:

English books:

The first one:

The second one:

Upon completion of the data collection process, the final result will resemble the image provided.

Training Model

The training of our model is effortlessly accomplished as Teachable Machine takes care of it automatically in the background. All that is required from us is to patiently await the results.

Click on the Train Model button and patiently wait for a few seconds until the training process is completed. It is important to note that you must keep your browser open on the Teachable Machine tab, as switching tabs will terminate the training session and require you to restart the training from the beginning.

Testing Classification Results in Browser (Preview Results)

Once the training of the model is complete, you will observe that the camera is activated, enabling you to directly evaluate the outcomes within your web browser.

Exporting Model

By exporting the model, we can utilize the pre-trained model in various contexts, such as gaming classification, scientific projects, and more.

In our scenario, we will export the model in Tensorflow format (.h5) to facilitate interaction with it using image processing packages and libraries like OpenCV or Pillow.

To proceed, simply click on the Export Model option depicted in the image displayed below.

We aim to utilize the TensorFlow (Keras) .h5 format to export our model for Python usage.

Select TensorFlow, Keras and click on Download my model button.

After a period of waiting, you will observe the presence of a compressed file containing two items:

keras_model.h5 (Our exported model)
labels.txt (text file for class names with IDs)

After opening the labels.txt file, you should find its contents to resemble the following:

0 Arabic
1 English

Classifying Images with Python

In the same directory, create a new Python file (e.g. named app.py).

It is crucial to ensure that both keras_model.h5 and labels.txt reside in the same directory (relative path) as the Python file.

Teachable Machine Package

Now, we need to install Teachable Machine for Python. The role of this package is to interact with the trained model by taking a captured image from camera by OpenCV, and converting it into tensors, so the model can classify it. In other words, it will handle the deep learning side for us in a few lines of code.

In your terminal, use this command to download teachable-machine package:

pip install teachable-machine

Accessing Camera using OpenCV

OpenCV offers hundreds of built-in methods and classes to be used in image processing and computer vision. In our project, accessing the camera with a view is required to start capturing the images.

Code of opening camera view using OpenCV:

import cv2 as cv

cap = cv.VideoCapture(0)

while True:
    _, img = cap.read()

    cv.imshow("Video Stream", img)

    cv.waitKey(1)

By running this code, you can open a view of your laptop's camera. It is important to know that the image frame is stored in img in our code, which will be operated as a matrix of numbers (represents pixels).

Capturing and Classifying Images

The next step is to capture an image from camera by using imwrite method in OpenCV. Then classifying the image using teachable-machine package.

Final code of capturing and classifying images:

from teachable_machine import TeachableMachine
import cv2 as cv

cap = cv.VideoCapture(0)
model = TeachableMachine(model_path="keras_model.h5",
                         labels_file_path="labels.txt")

image_path = "screenshot.jpg"

while True:
    _, img = cap.read()
    cv.imwrite(image_path, img)

    result = model.classify_image(image_path)

    print("class_index", result["class_index"])

    print("class_name:::", result["class_name"])

    print("class_confidence:", result["class_confidence"])

    print("predictions:", result["predictions"])

    cv.imshow("Video Stream", img)

    cv.waitKey(1)

The code will print different information about the trained model response such as:

Index of Class
Label/Name of Class
Accuracy (Confidence)
All predictions for other classes

Final Word

I hope that you've enjoyed the whole journey with image classification, if you're a developer trying to enhance the open-source tools, and you want to help me with developing and enhancing Teachable Machine Python package; PRs are welcomed at project repository.

Useful Links:

Happy Classification

SOLID Principles with Python | A story

Meqdad Darwish — Sat, 10 Jun 2023 12:01:18 +0000

To explore how to apply SOLID principles in practice, let's create a story.

The Beginning

Meet Mohammad, a smart person who wants to build a robust payment system.

First of all, Mohammad plans to create a class that has multiple responsibilities, such as adding items, calculating prices, creating a verification process, and accepting payments using different methods.

System initial code:

class Order:
    items = []
    quantities = []
    prices = []
    status = "open"

    def add_item(self, name, quantity, price):
        self.items.append(name)
        self.quantities.append(quantity)
        self.prices.append(price)

    def total_price(self):
        total = 0
        for i in range(len(self.prices)):
            total += self.quantities[i] * self.prices[i]
        return total

    def pay(self, payment_type, security_code):
        if payment_type == "debit":
            print("Processing payment type...")
            print(f"Verifying code: {security_code}.")
            self.status = "paid"
        elif payment_type == "credit":
            print("Processing payment type...")
            print(f"Verifying code: {security_code}.")
            self.status = "paid"
        else:
            raise Exception(f"#### Unknown type: {payment_type}.")

To use this class in code, Mohammad created this use case:

# Making orders
order = Order()
order.add_item("Clean Architecture Book, Uncle Bob", 1, 100)
order.add_item("Speaker", 1, 30)
order.add_item("HDMI Cable", 3, 10)
order.add_item("PIR Sensor", 2, 7)

print(order.total_price())

# Payment using Debit card
order.pay("debit", "123456789")

Output:

174
Processing payment type...
Verifying code: 123456789.

Everything works fine and Mohammad was satisfied for now.

SOLID Principles

Let's assume that "Uncle Clean", a consultant, is helping Mohammad implement the SOLID principles in his program.

SOLID stands for:

S: Single responsibility principle.
O: Open–closed principle.
L: Liskov substitution principle.
I: Interface segregation principle.
D: Dependency inversion principle.

1- Single Responsibility Principle (SRP)

Mohammad has asked himself about the responsibilities of the Order class in his code and has identified that it has multiple responsibilities, including adding items, calculating the total price, and handling payment details.

Uncle Clean in the Scene

Uncle Clean: Hi Mohammad, did you hear about SRP?

Mohammad: No, what is SRP?

Uncle Clean: The SRP (stands for Single Responsibility Principle) dictates that classes should have only a single reason to change. If your class contains multiple reasons for change; then it indicates that your code is tightly-coupled and harder to maintain.

Mohammad: What that means in my case?

Uncle Clean: What if a new customer requests a new payment method, such as Bitcoin or PayPal? In that case, you would need to modify the Order class. Therefore, it is recommended to separate the payment responsibility from the Order class.

After this advice, Mohammad changed his code to the following one:

class Order:
    items = []
    quantities = []
    prices = []
    status = "open"

    def add_item(self, name, quantity, price):
        self.items.append(name)
        self.quantities.append(quantity)
        self.prices.append(price)

    def total_price(self):
        total = 0
        for i in range(len(self.prices)):
            total += self.quantities[i] * self.prices[i]
        return total


class PaymentHandler:
    def pay_debit(self, order: Order, security_code):
        print("Processing Debit Card payment...")
        print(f"Verifying code: {security_code}.")
        order.status = "paid"

    def pay_credit(self, order: Order, security_code):
        print("Processing Credit Card payment...")
        print(f"Verifying code: {security_code}.")
        order.status = "paid"


# Making orders
order = Order()
order.add_item("Clean Architecture Book, Uncle Bob", 1, 100)
order.add_item("Speaker", 1, 30)
order.add_item("HDMI Cable", 3, 10)
order.add_item("PIR Sensor", 2, 7)

print(order.total_price())


# Payment using Credit card
payment_handler = PaymentHandler()
payment_handler.pay_credit(order, "543219876")

Output:

174
Processing Credit Card payment...
Verifying code: 543219876.

Uncle Clean: Great job, Mohammad! You're doing well. There is still room for optimization, such as creating a specific method for changing the order status, but for now, this is sufficient.

[For more details about SRP, take a look on Uncle Bob's blog from here]

2- Open/Closed Principle (OCP)

After a few days, a new client asked Mohammad if his program supported PayPal payments. Mohammad replied that this feature could be easily added.

To achieve that, Mohammad created a new method for PayPal, but he made a mistake while calling it, he used pay_debit instead of using pay_PayPal. Mohammad encountered additional comparable issues, which led him to realize that he was frequently modifying the PaymentHandler class each time a new payment feature was introduced.

Uncle Clean in the Scene

Uncle Clean: Hi Mohammad, After reviewing your recent difficulties when attempting to integrate new payment methods, my recommendation is to take the OCP into account as you continue to develop your code.

Mohammad: What is OCP?

Uncle Clean: OCP stands for Open/Closed Principle (OCP) which states that software entities (classes, functions, ...) should be open for extension but closed for modification. This means that when new requirements arise or changes need to be made, it should be possible to extend the behavior of the software entity without modifying its source code.

Now, after considering OCP into account, the new code became:

from abc import ABC, abstractmethod


class Order:
    items = []
    quantities = []
    prices = []
    status = "open"

    def add_item(self, name, quantity, price):
        self.items.append(name)
        self.quantities.append(quantity)
        self.prices.append(price)

    def total_price(self):
        total = 0
        for i in range(len(self.prices)):
            total += self.quantities[i] * self.prices[i]
        return total


class PaymentHandler(ABC):
    @abstractmethod
    def pay(self, order: Order, security_code):
        pass


class DebitPaymentHandler(PaymentHandler):
    def pay(self, order: Order, security_code):
        print("Processing Debit Card payment...")
        print(f"Verifying code: {security_code}.")
        order.status = "paid"


class CreditPaymentHandler(PaymentHandler):
    def pay(self, order: Order, security_code):
        print("Processing Credit Card payment...")
        print(f"Verifying code: {security_code}.")
        order.status = "paid"


class PayPalPaymentHandler(PaymentHandler):
    def pay(self, order: Order, security_code):
        print("Processing PayPal payment...")
        print(f"Verifying code: {security_code}.")
        order.status = "paid"


# Making orders
order = Order()
order.add_item("The Pragmatic Programmer Book, Andy Hunt", 1, 130)
order.add_item("micro:bit Controller", 5, 14)

print(order.total_price())


# Payment using PayPal

paypal_payment = PayPalPaymentHandler()
paypal_payment.pay(order, "543219876")

Output:

200
Processing PayPal payment...
Verifying code: 543219876.

3- Liskov Substitution Principle (LSP)

Next day after adding PayPal payment integration, a customer called Mohammad and told him that there is a big issue in the system! To pay using PayPal, you don't need a security code. Instead, you only need to provide your email address.

Mohammad considered changing the security_code to email, but this would create a new problem with other payment methods that require a security_code.

Uncle Clean in the Scene

Uncle Clean: Hey Mohammad, I heard about your last problem, why you won't use Liskov?

Mohammad: Hey Uncle, what you're talking about? Liskov???

Uncle Clean:

LSP is created by Prof. Barbara Liskov which stands for Liskov Substitution Principle, which states that objects of a superclass should be replaceable with objects of its subclasses without breaking the program's behavior.

The result of applying Liskov Substitution Principle on the codebase was:

from abc import ABC, abstractmethod


class Order:
    items = []
    quantities = []
    prices = []
    status = "open"

    def add_item(self, name, quantity, price):
        self.items.append(name)
        self.quantities.append(quantity)
        self.prices.append(price)

    def total_price(self):
        total = 0
        for i in range(len(self.prices)):
            total += self.quantities[i] * self.prices[i]
        return total


class PaymentHandler(ABC):
    @abstractmethod
    def pay(self, order: Order):
        pass


class DebitPaymentHandler(PaymentHandler):
    def __init__(self, security_code):
        self.security_code = security_code

    def pay(self, order: Order):
        print("Processing Debit Card payment...")
        print(f"Verifying code: {self.security_code}.")
        order.status = "paid"


class CreditPaymentHandler(PaymentHandler):
    def __init__(self, security_code):
        self.security_code = security_code

    def pay(self, order: Order):
        print("Processing Credit Card payment...")
        print(f"Verifying code: {self.security_code}.")
        order.status = "paid"


class PayPalPaymentHandler(PaymentHandler):
    def __init__(self, email) -> None:
        self.email = email

    def pay(self, order: Order):
        print("Processing PayPal payment...")
        print(f"Verifying email: {self.email}.")
        order.status = "paid"


# Making orders
order = Order()
order.add_item("The Pragmatic Programmer Book, Andy Hunt", 1, 130)
order.add_item("micro:bit Controller", 5, 14)

print(order.total_price())


# Payment using PayPal

paypal_payment = PayPalPaymentHandler("hi@customer.com")
paypal_payment.pay(order)

Output:

200
Processing PayPal payment...
Verifying email: hi@customer.com.

4- Interface Segregation Principle (ISP)

After a while, Mohammad contacted a cyber security expert to review the whole system and to submit a report that helps him to fix the vulnerabilities in the system for more protection.

One of the main points that this cyber security expert mentioned in his report: Adding 2FA ( Two-factor Authentication) to the system.

Based on that, Mohammad started to work on this feature by adding authentication method using SMS auth_2fa_sms in the PaymentHandler. And he implemented it to the other classes that are inheriting from the base class.

The code became as the following:

from abc import ABC, abstractmethod


class Order:
    items = []
    quantities = []
    prices = []
    status = "open"

    def add_item(self, name, quantity, price):
        self.items.append(name)
        self.quantities.append(quantity)
        self.prices.append(price)

    def total_price(self):
        total = 0
        for i in range(len(self.prices)):
            total += self.quantities[i] * self.prices[i]
        return total


class PaymentHandler(ABC):
    @abstractmethod
    def auth_2fa_sms(self, code):
        pass

    @abstractmethod
    def pay(self, order: Order):
        pass


class DebitPaymentHandler(PaymentHandler):
    def __init__(self, security_code):
        self.security_code = security_code
        self.verified = False

    def auth_2fa_sms(self, code):
        print(f"Verifying 2FA using SMS code: {code}")
        self.verified = True

    def pay(self, order: Order):
        if not self.verified:
            raise Exception("Not authenticated")
        print("Processing Debit Card payment...")
        print(f"Verifying code: {self.security_code}.")
        order.status = "paid"


class CreditPaymentHandler(PaymentHandler):
    def __init__(self, security_code):
        self.security_code = security_code

    def auth_2fa_sms(self, code):
        # It is also a violation for Liskov Substitution principle
        raise Exception(
            "Credit card payment doesn't support SMS code authentication.")

    def pay(self, order: Order):
        print("Processing Credit Card payment...")
        print(f"Verifying code: {self.security_code}.")
        order.status = "paid"


class PayPalPaymentHandler(PaymentHandler):
    def __init__(self, email) -> None:
        self.email = email

    def auth_2fa_sms(self, code):
        print(f"Verifying 2FA using SMS code: {code}")
        self.verified = True

    def pay(self, order: Order):
        if not self.verified:
            raise Exception("Not authenticated")
        print("Processing PayPal payment...")
        print(f"Verifying email: {self.email}.")
        order.status = "paid"


# Making orders
order = Order()
order.add_item("Head First Object Oriented Analysis and Design Book", 1, 76)
order.add_item("Raspberry Pi Camera v2", 2, 40)

print(order.total_price())


# Payment using Debit Card

debit_payment = DebitPaymentHandler("67891")
debit_payment.auth_2fa_sms("54321")
debit_payment.pay(order)

As in the above code, when the user pay for his stuff using debit card, it should be authenticated before payment process, and the expected result will be as the following:

156
Verifying 2FA using SMS code: 54321
Processing Debit Card payment...
Verifying code: 67891.

Uncle Clean in the Scene

Uncle Clean: Hello Abo Ehmaid (a nickname for Mohammad), I heard that you're securing your payment software, a good news!
But what will happen if the user start using credit card?

Mohammad: Hi uncle, I know that credit card doesn't support 2FA using SMS verification, so I added an exception for this error. But I think it should be implemented in a better way. What do you advise me to do?

Uncle Clean: I suggest on you to use interface segregation.

Mohammad: ?????

Uncle Clean: Interface Segregation Principle (ISP) means: Clients should not be forced to depend on interfaces they do not use (SMS 2FA with credit card in your case).

Mohammad: I'll read about it and fix that.

After reading using different resources about ISP, Mohammad ended up with this modification:

from abc import ABC, abstractmethod


class Order:
    items = []
    quantities = []
    prices = []
    status = "open"

    def add_item(self, name, quantity, price):
        self.items.append(name)
        self.quantities.append(quantity)
        self.prices.append(price)

    def total_price(self):
        total = 0
        for i in range(len(self.prices)):
            total += self.quantities[i] * self.prices[i]
        return total


class PaymentHandler(ABC):
    @abstractmethod
    def pay(self, order: Order):
        pass


class SMSPaymentAuthHandler(PaymentHandler):
    @abstractmethod
    def auth_2fa_sms(self, code):
        pass


class DebitPaymentHandler(SMSPaymentAuthHandler):
    def __init__(self, security_code):
        self.security_code = security_code
        self.verified = False

    def auth_2fa_sms(self, code):
        print(f"Verifying 2FA using SMS code: {code}")
        self.verified = True

    def pay(self, order: Order):
        if not self.verified:
            raise Exception("Not authenticated")
        print("Processing Debit Card payment...")
        print(f"Verifying code: {self.security_code}.")
        order.status = "paid"


class CreditPaymentHandler(PaymentHandler):
    def __init__(self, security_code):
        self.security_code = security_code

    def pay(self, order: Order):
        print("Processing Credit Card payment...")
        print(f"Verifying code: {self.security_code}.")
        order.status = "paid"


class PayPalPaymentHandler(SMSPaymentAuthHandler):
    def __init__(self, email) -> None:
        self.email = email

    def auth_2fa_sms(self, code):
        print(f"Verifying 2FA using SMS code: {code}")
        self.verified = True

    def pay(self, order: Order):
        if not self.verified:
            raise Exception("Not authenticated")
        print("Processing PayPal payment...")
        print(f"Verifying email: {self.email}.")
        order.status = "paid"


# Making orders
order = Order()
order.add_item("Head First Object Oriented Analysis and Design Book", 1, 76)
order.add_item("Raspberry Pi Camera v2", 2, 40)

print(order.total_price())


# Payment using Debit Card

debit_payment = DebitPaymentHandler("67891")
debit_payment.auth_2fa_sms("54321")
debit_payment.pay(order)

print("#"*20)

# Payment using Credit Card

credit_payment = CreditPaymentHandler("97531")
credit_payment.pay(order)

Output:

156
Verifying 2FA using SMS code: 54321
Processing Debit Card payment...
Verifying code: 67891.
####################
Processing Credit Card payment...
Verifying code: 97531.

Composition over Inheritance

Note: Composition over Inheritance is not one of the SOLID principles, but it is worth mentioning in the story context because it is more efficient in this case.

Uncle Clean in the Scene

Uncle Clean: Hi Mohammad, you did a good job by separating the interfaces.

Mohammad: Thanks uncle, your follow up and advices for me are highly appreciated.

Uncle Clean: Are you familiar with various types of two-factor authentication (2FA) methods?

Mohammad: Yes, I am. I know different types, such as SMS-based, authenticator apps, Email-based, and more.

Uncle Clean: That's great to hear! Now, let's say you encounter a situation where you need to incorporate multiple 2FA methods for the same payment method, like adding an authenticator app alongside SMS-based authentication for PayPal. How would you approach this? Additionally, do you believe that solely inheriting from the 2FA classes would be sufficient for your payment methods? I'm referring to the need for additional features to be added to your payment classes, beyond inheriting from the 2FA classes.

Mohammad: Well, there are situations where I could create additional classes to inherit the necessary features for payment methods. However, I'm concerned about the long-term practicality if I keep adding more and more features. Uncle, I would like to hear your thoughts on this matter.

Uncle Clean: In certain scenarios, employing inheritance may not be the most practical or efficient approach to transferring features from one class to another. In such cases, it is advisable to consider the principle of Composition over Inheritance (CoI) as a better alternative.

Mohammad: In simple words, what do we mean by that?

Uncle Clean: Composition over inheritance is a principle in OOP that suggests favoring composition, or building objects by combining simpler components, over inheritance, where objects inherit properties and behaviors from parent classes. With composition, we can combine smaller, modular components for greater flexibility and code reusability. It reduces tight coupling, avoids fragility, and improves code readability and maintainability. Embracing composition will result in more efficient and adaptable code.

Mohammad: Interesting! I will do my best to apply that. Thanks uncle.

Uncle Clean: Habibi teslam. (a greeting word in Arabic)

After reviewing different examples, Mohammad changed his codebase to the following one: (Added SMSAuthorizer and created an object in debit card class)

from abc import ABC, abstractmethod


class Order:
    items = []
    quantities = []
    prices = []
    status = "open"

    def add_item(self, name, quantity, price):
        self.items.append(name)
        self.quantities.append(quantity)
        self.prices.append(price)

    def total_price(self):
        total = 0
        for i in range(len(self.prices)):
            total += self.quantities[i] * self.prices[i]
        return total


class SMSAuthorizer:

    authorized = False

    def verify_code(self, code):
        print(f"Verifying code {code}")
        self.authorized = True

    def is_authorized(self):
        return self.authorized


class PaymentHandler(ABC):
    @abstractmethod
    def pay(self, order: Order):
        pass


class DebitPaymentHandler(PaymentHandler):
    def __init__(self, security_code, authorizer: SMSAuthorizer):
        self.authorizer = authorizer
        self.security_code = security_code

    def pay(self, order: Order):
        if not self.authorizer.is_authorized():
            raise Exception("Not authenticated")
        print("Processing Debit Card payment...")
        print(f"Verifying code: {self.security_code}.")
        order.status = "paid"


class CreditPaymentHandler(PaymentHandler):
    def __init__(self, security_code):
        self.security_code = security_code

    def pay(self, order: Order):
        print("Processing Credit Card payment...")
        print(f"Verifying code: {self.security_code}.")
        order.status = "paid"


class PayPalPaymentHandler(PaymentHandler):
    def __init__(self, email, authorizer: SMSAuthorizer) -> None:
        self.authorizer = authorizer
        self.email = email

    def pay(self, order: Order):
        if not self.authorizer.is_authorized():
            raise Exception("Not authenticated")
        print("Processing PayPal payment...")
        print(f"Verifying email: {self.email}.")
        order.status = "paid"


# Making orders
order = Order()
order.add_item("Head First Object Oriented Analysis and Design Book", 1, 76)
order.add_item("Raspberry Pi Camera v2", 2, 40)

print(order.total_price())


# SMS Authorizer
sms_authorizer = SMSAuthorizer()

# Payment using Debit Card

debit_payment = DebitPaymentHandler("67891", sms_authorizer)
sms_authorizer.verify_code("24682")
debit_payment.pay(order)

Output:

156
Verifying code 24682
Processing Debit Card payment...
Verifying code: 67891.

5- Dependency Inversion Principle (DIP)

At this point, Mohammad was smart enough to read about DIP principle 🤣, where he called uncle Clean and introduced DIP for him.

Uncle Clean in the Scene

Mohammad: Hi uncle, I want to know that I worked on DIP during last week.

Uncle Clean: What do you mean by DIP?

Mohammad: I can't imagine that, I am explaining a tech topic for uncle Clean! 😊

Anyway, DIP stands for the Dependency Inversion Principle. The principle is about removing dependencies from high-level code to low-level code by creating interfaces, such as Authorizer in my case. As a result, both high-level and low-level code depend on these interfaces.

Uncle Clean: Great job Mohammad, you're amazing. Based on that, what you've changed in your code?

Mohammad: I created Authorizer, where I pass it to the suitable payment methods, and this gives me the ability to pass different authorization types if they are subclasses from the Authorizer interface. Here is the code below:

from abc import ABC, abstractmethod


class Order:
    items = []
    quantities = []
    prices = []
    status = "open"

    def add_item(self, name, quantity, price):
        self.items.append(name)
        self.quantities.append(quantity)
        self.prices.append(price)

    def total_price(self):
        total = 0
        for i in range(len(self.prices)):
            total += self.quantities[i] * self.prices[i]
        return total


class Authorizer(ABC):
    @abstractmethod
    def is_authorized(self):
        pass


class SMSAuthorizer(Authorizer):

    authorized = False

    def verify_code(self, code):
        print(f"Verifying code {code}")
        self.authorized = True

    def is_authorized(self):
        return self.authorized


class PaymentHandler(ABC):
    @abstractmethod
    def pay(self, order: Order):
        pass


class DebitPaymentHandler(PaymentHandler):
    def __init__(self, security_code, authorizer: Authorizer):
        self.authorizer = authorizer
        self.security_code = security_code

    def pay(self, order: Order):
        if not self.authorizer.is_authorized():
            raise Exception("Not authenticated")
        print("Processing Debit Card payment...")
        print(f"Verifying code: {self.security_code}.")
        order.status = "paid"


class CreditPaymentHandler(PaymentHandler):
    def __init__(self, security_code):
        self.security_code = security_code

    def pay(self, order: Order):
        print("Processing Credit Card payment...")
        print(f"Verifying code: {self.security_code}.")
        order.status = "paid"


class PayPalPaymentHandler(PaymentHandler):
    def __init__(self, email, authorizer: Authorizer) -> None:
        self.authorizer = authorizer
        self.email = email

    def pay(self, order: Order):
        if not self.authorizer.is_authorized():
            raise Exception("Not authenticated")
        print("Processing PayPal payment...")
        print(f"Verifying email: {self.email}.")
        order.status = "paid"


# Making orders
order = Order()
order.add_item("Head First Object Oriented Analysis and Design Book", 1, 76)
order.add_item("Raspberry Pi Camera v2", 2, 40)

print(order.total_price())


# SMS Authorizer
sms_authorizer = SMSAuthorizer()

# Payment using Debit Card

debit_payment = DebitPaymentHandler("67891", sms_authorizer)
sms_authorizer.verify_code("24682")
debit_payment.pay(order)

Uncle Clean: Wow, that is a great progress. What if we want to add a reCAPTCHA authentication method? Can the Dependency Inversion Principle be helpful here?

Mohammad: Sure, I added that to my code, where I created a concrete class reCAPTCHA_Authorizer from Authorizer abstract class, and I passed it to PayPal payment class. So now I can use different authorization and authentication methods to the same payment method class. Look at this code below:

from abc import ABC, abstractmethod


class Order:
    items = []
    quantities = []
    prices = []
    status = "open"

    def add_item(self, name, quantity, price):
        self.items.append(name)
        self.quantities.append(quantity)
        self.prices.append(price)

    def total_price(self):
        total = 0
        for i in range(len(self.prices)):
            total += self.quantities[i] * self.prices[i]
        return total


class Authorizer(ABC):
    @abstractmethod
    def is_authorized(self):
        pass


class reCAPTCHA_Authorizer(Authorizer):

    authorized = False

    def verify_reCAPTCHA(self, user_selections):
        print(f"Verifying reCAPTCHA {user_selections}")
        print("User passed `I'm not a robot` test")
        self.authorized = True

    def is_authorized(self):
        return self.authorized


class SMSAuthorizer(Authorizer):

    authorized = False

    def verify_code(self, code):
        print(f"Verifying code {code}")
        self.authorized = True

    def is_authorized(self):
        return self.authorized


class PaymentHandler(ABC):
    @abstractmethod
    def pay(self, order: Order):
        pass


class DebitPaymentHandler(PaymentHandler):
    def __init__(self, security_code, authorizer: Authorizer):
        self.authorizer = authorizer
        self.security_code = security_code

    def pay(self, order: Order):
        if not self.authorizer.is_authorized():
            raise Exception("Not authenticated")
        print("Processing Debit Card payment...")
        print(f"Verifying code: {self.security_code}.")
        order.status = "paid"


class CreditPaymentHandler(PaymentHandler):
    def __init__(self, security_code):
        self.security_code = security_code

    def pay(self, order: Order):
        print("Processing Credit Card payment...")
        print(f"Verifying code: {self.security_code}.")
        order.status = "paid"


class PayPalPaymentHandler(PaymentHandler):
    def __init__(self, email, authorizer: Authorizer) -> None:
        self.authorizer = authorizer
        self.email = email

    def pay(self, order: Order):
        if not self.authorizer.is_authorized():
            raise Exception("Not authenticated")
        print("Processing PayPal payment...")
        print(f"Verifying email: {self.email}.")
        order.status = "paid"


# Making orders
order = Order()
order.add_item("Domain Driven Design Book, by Eric Evans", 2, 111)
order.add_item("Raspberry Pi Pico", 3, 15)

print(order.total_price())


# reCAPTCHA Authorizer
reCAPTCHA_authorizer = reCAPTCHA_Authorizer()

# Payment using PayPal

paypal_payment = PayPalPaymentHandler("hi@customer.com", reCAPTCHA_authorizer)
reCAPTCHA_authorizer.verify_reCAPTCHA("User Selections: 1,4,7")
paypal_payment.pay(order)

Output:

267
Verifying reCAPTCHA User Selections: 1,4,7
User passed `I'm not a robot` test
Processing PayPal payment...
Verifying email: hi@customer.com.

Final Thoughts

Firstly, I want to thank my characters in the story, Mohammad and uncle Clean.

Secondly, SOLID principles aim to make software more flexible, maintainable, scalable, and testable by reducing code dependencies and making designs easier to understand, maintain, and extend.

However, SOLID principles are not always applicable in every situation, and it is important to use them correctly and embrace the soul behind the SOLID rules. 😊

By following these guidelines, developers can create cleaner code that is easier to modify and understand, leading to better collaboration among team members and more robust, flexible, and reusable software.

Finally, happy coding.

Useful links:

GitHub version of article
Files of Code Samples
Inspired from ArjanCodes video
Want more explanation? Ask ChatGPT, Bard, Bing Chat and other guys.