DEV Community: Ajay Balakumaran

Code Your Diagrams: Automate Architecture with Python's Diagrams Library

Ajay Balakumaran — Mon, 06 Jan 2025 06:40:26 +0000

Introduction

In the realm of modern infrastructure, where cloud services and microservices reign supreme, managing and visualizing complex architectures is more critical than ever.

Gone are the days of manually creating and updating architecture diagrams. With the Diagrams Python library, you can generate dynamic, code-driven diagrams that evolve alongside your infrastructure. A few lines of Python are all it takes to visualize cloud architectures, network topologies, or microservice interactions. Diagrams ensure your system documentation stays accurate and up-to-date, whether you’re managing multi-cloud deployments, Kubernetes clusters, or on-premise solutions. It’s an effortless way to keep your architecture in sync with your codebase.

In this post, we'll explore the capabilities of the Diagrams library, showcase how to create High-Level Designs (HLDs) for cloud infrastructure, and automate the process of creating architecture diagrams.

Why Use Diagrams Python Library?

Automation: Generate architecture diagrams directly from your code, ensuring they stay up-to-date with the evolving system.
Programmatic Control: Diagrams allows you to define your infrastructure visually with Python, offering fine control over how elements are represented.
Supports Multiple Cloud Providers: The library supports AWS, Azure, GCP, and on-premise systems, making it a versatile tool for visualizing multi-cloud and hybrid architectures.
Scalable: From small projects to large distributed systems, Diagrams can handle various levels of complexity.

Supported Providers

OnPrem
AWS
Azure
GCP
IBM
Kubernetes (K8s)
AlibabaCloud
OCI (Oracle Cloud Infrastructure)
OpenStack
Firebase
DigitalOcean
Elastic
Outscale
Generic
Programming
SaaS
C4 Model
Custom

Getting Started with Diagrams

To get started with Diagrams, you’ll need to install the library and set up your environment.

Step 0: Prerequisites

To create diagrams using the Diagrams Python library on MacOS/Windows, you'll need to install Graphviz first. Graphviz is the tool that the Diagrams library uses to generate the visual representations of your infrastructure.

Mac

If you're on macOS, the easiest way to install Graphviz is using Homebrew:

brew install graphviz

Windows

If you're on Windows, follow the below steps

Download the Graphviz installer from the official website Graphviz Download Page.
Run the installer and follow the installation steps.
During installation, make sure to check the option that adds Graphviz to your system’s PATH.

Step 1: Installation

pip install diagrams

Step 2: Your First Diagram

Let’s create a simple diagram that represents a basic web architecture on AWS.

from diagrams import Diagram
from diagrams.aws.compute import EC2
from diagrams.aws.network import ELB
from diagrams.aws.database import RDS

with Diagram("Simple AWS Architecture", show=False):
    lb = ELB("Load Balancer")
    web = EC2("Web Server")
    db = RDS("Database")

    lb >> web >> db

With this minimal code, you can visualize how traffic flows from the Load Balancer to the Web Server, and then to the Database. That’s the power of the Diagrams library: it's quick, intuitive, and highly customizable. And this is just the beginning—there are many more advanced features and components you can leverage, which we'll explore in the following sections.

Advanced Features

Grouping Components (Clustering)

You can group components into clusters to represent different tiers or logical groupings within your architecture.

from diagrams import Cluster, Diagram
from diagrams.aws.compute import EC2
from diagrams.aws.network import ELB
from diagrams.aws.database import RDS

with Diagram("AWS Architecture with Clustering", show=False):
    with Cluster("Web Tier"):
        lb = ELB("Load Balancer")
        web_servers = [EC2("Web 1"), EC2("Web 2")]

    with Cluster("Database Tier"):
        db_primary = RDS("Primary DB")
        db_replica = RDS("Replica DB")

    lb >> web_servers >> db_primary
    db_primary >> db_replica

We use Cluster() to group web servers and databases, making the diagram easier to understand by visualizing tiers separately.

Customizing Components

Diagrams allow you to add custom labels, colors, and even custom images to represent specific components. For example, if you want to represent a custom service, you can include external images from local or even from remote.

Using a Custom Icon from a Local Source

If you have an icon saved locally (for example, a custom_icon.png file), you can use it to represent your custom component in the diagram. The code below shows how to add a custom icon from your local filesystem.

from diagrams.custom import Custom
with Diagram("Custom Service Architecture", show=False):
    custom_service = Custom("My Custom Service", "./custom_icon.png")

./custom_icon.png is the path to your local image file.

Using a Custom Icon from a Remote Source

Similarly, you can use an image from a remote source. Here's how you can download an image from a URL and use it in your diagram.

You can also use custom icons from a remote URL by giving the remote path to the files.

from diagrams import Diagram, Cluster
from diagrams.custom import Custom
from urllib.request import urlretrieve

with Diagram("Custom with remote icons", show=False, filename="custom_remote", direction="LR"):

  # download the icon image file
  diagrams_url = "https://github.com/mingrammer/diagrams/raw/master/assets/img/diagrams.png"
  diagrams_icon = "diagrams.png"
  urlretrieve(diagrams_url, diagrams_icon)

  diagrams = Custom("Diagrams", diagrams_icon)

This allows for even more flexibility in designing architecture that suits your organization's needs.

Combining Multi-Cloud and On-Premise Architectures

We can also use a combination of on-premise systems and cloud infrastructure, Diagrams makes it easy to combine these elements into a single view. You can visualize hybrid architectures seamlessly.

from diagrams import Diagram, Cluster, Edge
from diagrams.aws.compute import EC2
from diagrams.aws.network import ELB
from diagrams.aws.database import RDS
from diagrams.gcp.compute import GCE
from diagrams.azure.compute import VM
from diagrams.onprem.compute import Server
from diagrams.onprem.client import User
from diagrams.generic.network import Firewall, Switch, Router
from diagrams.generic.storage import Storage  

with Diagram("Multi-Cloud and On-Prem Architecture", show=False):

    # On-premise infrastructure
    with Cluster("On-Premise Data Center"):
        users = User("Users")
        firewall = Firewall("Firewall")
        router = Router("Router")
        switch = Switch("Switch")
        onprem_server = Server("Local Server")
        storage = Storage("Storage")

        users >> firewall >> router >> switch >> onprem_server
        onprem_server >> storage

    # AWS infrastructure
    with Cluster("AWS Cloud"):
        aws_lb = ELB("Load Balancer")
        aws_ec2 = EC2("App Servers")
        aws_rds = RDS("Database")

        aws_lb >> aws_ec2 >> aws_rds

    # GCP infrastructure
    with Cluster("GCP Cloud"):
        gcp_gce = GCE("Compute Engine")
        gcp_storage = Storage("GCP Storage")

        gcp_gce >> gcp_storage

    # Azure infrastructure
    with Cluster("Azure Cloud"):
        azure_vm = VM("Virtual Machines")
        azure_storage = Storage("Azure Storage")

        azure_vm >> azure_storage

    # Connecting on-prem to cloud components via VPN
    router >> Edge(label="VPN") >> [aws_lb, gcp_gce, azure_vm]

Challenges and Limitations

Although Diagrams is a powerful tool, there are some challenges:

Performance: Generating very large diagrams with hundreds of components can be slow.
Customization Limitations: While Diagrams offers a wide range of predefined components, adding highly customized elements might require extra work.
Static Output: Diagrams generate static images. If you need interactive or real-time diagrams, you may need to integrate them with other tools.

Conclusion

The Diagrams Python library is a fantastic tool for automating the creation of infrastructure diagrams. By integrating it into your workflows, you can generate architecture diagrams dynamically as your infrastructure changes. Whether you’re documenting your cloud infrastructure or illustrating complex microservice architectures, Diagrams offers a powerful, programmatic way to visualize your systems

GitHub Repository

You can find the complete source code for the examples in this blog on my GitHub:

My Diagrams Code Repository

Reference

Diagrams: https://diagrams.mingrammer.com/docs/getting-started/installation

Disclaimer:

This is a personal blog. The views and opinions expressed here are only those of the author and do not represent those of any organization or any individual with whom the author may be associated, professionally or personally.

Python 3.13: The Gateway to High-Performance Multithreading Without GIL

Ajay Balakumaran — Mon, 06 Jan 2025 06:40:14 +0000

Introduction

Python has long been known for its ease of use and versatility, but one topic that has sparked much discussion in the Python community is the Global Interpreter Lock (GIL). The GIL has been both a safeguard and a bottleneck for Python's concurrency model, especially for CPU-bound tasks that could otherwise take advantage of multiple CPU cores. With the release of Python 3.13, however, Python developers have a groundbreaking new option: the ability to disable the GIL. This blog will explore what the GIL is, why it has been an obstacle for performance in multithreading, and how to detect and disable the GIL in Python 3.13 to unlock true multithreading performance.

What is Global Interpreter Lock (GIL)

The Global Interpreter Lock (GIL) is a mutex that protects access to Python objects, preventing multiple native threads from executing Python bytecode at once. This ensures thread safety for Python programs but at the cost of concurrent execution. The GIL makes Python threads more efficient for I/O-bound tasks but limits their performance for CPU-bound tasks.

Why the GIL is a Bottleneck for Multithreading

Python's GIL allows only one thread to execute simultaneously, even in multithreaded programs. While this is fine for I/O-bound tasks where the program is waiting for input/output operations, it severely limits performance for CPU-bound tasks like number crunching, data analysis, or image processing.

Python 3.13: Unlocking Multithreading with GIL Disabled

With Python 3.13, developers have the option to disable the GIL during the Python build process. However, disabling the GIL is not available in pre-built Python distributions. Instead, you must compile Python 3.13 from source with the --disable-gil option.

This new option opens the door for true parallelism in CPU-bound multithreaded tasks, allowing threads to execute in parallel across multiple cores.

Prerequisites for Using Python 3.13 Without GIL

Python 3.13 Source Code: Disabling the GIL is not available in standard pre-built binaries. You must build Python 3.13 from the source with the --disable-gil flag.
Multi-Core CPU: You need a multi-core CPU to benefit from true multithreading, as threads will now run in parallel across multiple cores.

Compiling Python 3.13 with GIL Disabled

To disable the GIL using the -X gil=0 flag, you need to compile Python from source with the --disable-gil flag enabled. Here's how to do that

Step-by-Step

Download Python 3.13 Source Code You first need to download the Python 3.13 source code tarball from the official Python website. This is because the pre-built binaries (like the ones you download directly from python.org) are not compiled with support for disabling the GIL. You can download it using the web browser or using wget or even using curl in your terminal

wget https://www.python.org/ftp/python/3.13.0/Python-3.13.0.tgz

Extract the Source:

tar -xf Python-3.13.0.tgz
cd Python-3.13.0

Configure the build with --disable-gil You need to configure Python with --disable-gil to support the option of disabling the GIL.

./configure --disable-gil

Compile and install Python:

make
sudo make altinstall

If altinstall step fails, then Re-run the configure Command with --prefix

./configure --disable-gil --prefix=$HOME/python3.13

Run make altinstall in the Specified Directory Then, run the make altinstall command

make altinstall

How to Detect GIL in Python 3.13

In Python 3.13, you can check whether the GIL is enabled or disabled using the sys._is_gil_enabled() function.

import sys

def check_gil_status():
    if sys.version_info >= (3, 13):
        status = sys._is_gil_enabled()
        if status:
            print("GIL is currently enabled.")
        else:
            print("GIL is currently disabled.")
    else:
        print("Python version does not support GIL status detection.")

check_gil_status()

Hands-On: Python Multithreading with GIL vs GIL-Free

The following Python code was developed to assess the performance gains when disabling the GIL in Python 3.13. The script executes eight threads concurrently, each tasked with calculating the prime factors of large numbers. By leveraging true parallelism, the code highlights the enhanced performance achieved without the GIL.

#!/usr/bin/env python3
import sys
import sysconfig
import time
from threading import Thread
from multiprocessing import Process


# Decorator to measure execution time of functions
def calculate_execution_time(func):
    def wrapper(*args, **kwargs):
        start_time = time.perf_counter()
        result = func(*args, **kwargs)
        end_time = time.perf_counter()
        execution_time = end_time - start_time
        print(f"{func.__name__} took {execution_time:.4f} seconds.")
        return result

    return wrapper


# Compute-intensive task: Iterative Fibonacci calculation
def compute_fibonacci(n):
    """Compute Fibonacci number for a given n iteratively."""
    a, b = 0, 1
    for _ in range(n):
        a, b = b, a + b
    return a


# Single-threaded task execution
@calculate_execution_time
def run_single_threaded(nums):
    for num in nums:
        compute_fibonacci(num)


# Multi-threaded task execution
@calculate_execution_time
def run_multi_threaded(nums):
    threads = [Thread(target=compute_fibonacci, args=(num,)) for num in nums]
    for thread in threads:
        thread.start()
    for thread in threads:
        thread.join()


# Multi-processing task execution
@calculate_execution_time
def run_multi_processing(nums):
    processes = [Process(target=compute_fibonacci, args=(num,)) for num in nums]
    for process in processes:
        process.start()
    for process in processes:
        process.join()


# Main execution function
def main():
    # Check Python version and GIL status for Python 3.13+
    print(f"Python Version: {sys.version}")

    py_version = float(".".join(sys.version.split()[0].split(".")[:2]))
    status = sysconfig.get_config_var("Py_GIL_DISABLED")

    if py_version >= 3.13:
        status = sys._is_gil_enabled()

    if status is None:
        print("GIL cannot be disabled for Python <= 3.12")
    elif status == 0:
        print("GIL is currently disabled")
    elif status == 1:
        print("GIL is currently active")

    # Run tasks on the same input size for comparison
    nums = [300000] * 8

    print("\nRunning Single-Threaded Task:")
    run_single_threaded(nums)

    print("\nRunning Multi-Threaded Task:")
    run_multi_threaded(nums)

    print("\nRunning Multi-Processing Task:")
    run_multi_processing(nums)


if __name__ == "__main__":
    main()

Analysis:

## Python 3.13 with GIL Disabled
Python Version: 3.13.0 experimental free-threading build (main, Oct 14 2024, 17:09:28) [Clang 14.0.0 (clang-1400.0.29.202)]
GIL is currently disabled

Running Single-Threaded Task:
run_single_threaded took 8.6587 seconds.

Running Multi-Threaded Task:
run_multi_threaded took 1.3885 seconds.

Running Multi-Processing Task:
run_multi_processing took 1.5953 seconds.

## Python 3.13 with GIL Enabled
Python Version: 3.13.0 experimental free-threading build (main, Oct 14 2024, 17:09:28) [Clang 14.0.0 (clang-1400.0.29.202)]
GIL is currently active

Running Single-Threaded Task:
run_single_threaded took 8.7108 seconds.

Running Multi-Threaded Task:
run_multi_threaded took 8.6645 seconds.

Running Multi-Processing Task:
run_multi_processing took 1.4530 seconds.

## Python 3.12 
Python Version: 3.12.6 (main, Sep  7 2024, 19:30:10) [Clang 14.0.0 (clang-1400.0.29.202)]
GIL cannot be disabled for Python <= 3.12

Running Single-Threaded Task:
run_single_threaded took 8.7004 seconds.

Running Multi-Threaded Task:
run_multi_threaded took 8.6297 seconds.

Running Multi-Processing Task:
run_multi_processing took 1.4876 seconds.

Multi-threaded performance: The real benefit of disabling the GIL is evident in the multi-threaded scenario:

With GIL disabled (3.13), the execution time is 1.5703 seconds.
With GIL enabled (3.13), the execution time is 8.5901 seconds.
Result: Disabling the GIL resulted in a performance improvement of approximately 81.7% for multi-threaded tasks.

The chart clearly demonstrates that disabling the GIL in Python 3.13 leads to a substantial performance boost for multi-threaded CPU-bound tasks, allowing Python to efficiently utilize multiple CPU cores in parallel. While single-threaded and multi-processing performance remains largely unaffected, multi-threaded performance shows a significant improvement, making Python 3.13 a game-changer for CPU-intensive applications that rely on multi-threading.

However, Python versions prior to 3.13 do not support disabling the GIL, which explains why their multi-threaded performance remains similar to that of Python 3.13 with the GIL enabled. This limitation in earlier versions continues to restrict Python's ability to fully exploit multi-threading for CPU-bound tasks.

Key Considerations Before Disabling the GIL

Disabling the Global Interpreter Lock (GIL) in Python 3.13 can unlock significant performance improvements in multi-threaded CPU-bound tasks. However, there are several important factors to consider before doing so:

Thread Safety: Without the GIL, you must manually handle thread safety using locks or other synchronization mechanisms to prevent race conditions in your code.
Potential Performance Degradation: Fine-grained locking can introduce contention, which may degrade performance in single-threaded or I/O-bound tasks that previously benefited from the GIL.
Compatibility with Third-Party Libraries: Many C extensions and libraries assume the presence of the GIL for thread safety. Disabling the GIL might require updates to these libraries to ensure they work correctly in a multithreaded environment.
Complex Memory Management: Disabling the GIL introduces more complexity in memory management, requiring thread-safe memory handling which can increase the risk of bugs and errors.
I/O-bound Tasks: Disabling the GIL provides limited benefits for I/O-bound tasks, where non-blocking I/O mechanisms like asyncio might be more effective.
Difficulty in Debugging: Without the GIL, debugging multithreaded applications can become more challenging due to the increased likelihood of race conditions and deadlocks.
Higher Memory Usage: Using locks and managing thread states without the GIL can increase memory consumption, particularly in multi-threaded applications.
Embedded Systems: Disabling the GIL might complicate Python's integration with multi-threaded environments in embedded systems, requiring more effort for effective integration.
Lock Contention: In some cases, disabling the GIL can lead to lock contention between threads, which might reduce the expected performance improvements.

GitHub Repository

You can find the complete source code for the examples in this blog on my GitHub:

Python GIL Performance Analysis

Disclaimer:

Roadmap to Problem-Solving Excellence

Ajay Balakumaran — Sun, 13 Aug 2023 06:55:56 +0000

Introduction

Selecting the right data structure and algorithm is a critical aspect of problem-solving in computer science. Selecting the DS and Algo in CI or in our project use cases is difficult. The below cheat sheet aims to guide us through the process of making decisions while selecting a proper data structure and algorithm.

Understanding the characteristics of the problem and analyzing its requirements are key factors in identifying the best approach. The cheat sheet will provide you with a structured framework to match problem traits with the strengths of various data structures and algorithms.

We can gain insights from the below cheatsheet

Recognize the problems that can be efficiently solved using arrays, linked lists, stacks, queues, trees, graphs, and more.
Identifying when to employ search algorithms like binary search, linear search, or depth-first search (DFS) and breadth-first search (BFS).
Knowing when to apply dynamic programming, backtracking, or greedy algorithms to tackle specific problem types.
Distinguishing between problems suited for sliding window, two pointers, or hash table techniques.

This cheat sheet is your trusty companion, designed to simplify decision-making and empower you to make informed choices.

Takeaway:

Problem-solving is an iterative process; the more you practice, the sharper your skills will become.
Familiarize yourself with different algorithm paradigms like divide and conquer, dynamic programming, greedy algorithms, and backtracking. Recognize when to apply each technique for optimal solutions.
The world of data structures and algorithms is vast. Embrace a growth mindset and continue learning advanced concepts to improve your problem-solving skills continually.
Stay curious, don't shy away from challenging problems, and enjoy the thrill of cracking them.
Revisit key concepts, practice coding problems, and gain confidence in interviews.

Disclaimer:

Efficient Frequency Estimation with Count-Min Sketch: A Memory-Saving Approach in Python

Ajay Balakumaran — Wed, 26 Jul 2023 05:50:57 +0000

Table of Content

Introduction
Frequency count
What is CM Sketch
Why is CM Sketch is important?
Super Linear vs Linear vs Sub Linear
Why not hash tables?
Problems in CM Sketch ?
Unveiling the Inner Workings
From Theory to Action: Implementation Unleashed
In Python
Memory Comparison
Summary of Key Points
Takeaway
Applications
References

Introduction

Count-Min Sketch is a popular probabilistic data structure used for approximate frequency estimation of elements in a data stream. It provides a memory-efficient (sub-linear) way to estimate the frequency of elements without storing the entire dataset. In this blog, we'll explore the concept of Count-Min Sketch and learn how to implement and use it in Python.

Frequency count

Frequency count, or frequency analysis, is a technique in data analysis that involves determining the number of occurrences of distinct elements in a dataset or data stream.
It provides valuable insights into the distribution and popularity of elements within the data. By counting how often each element appears, frequency counts help uncover patterns, identify trends, and reveal the most common or significant elements in the dataset.
This technique finds applications in various domains, including text analysis, data mining, web analytics, market research, network traffic analysis, and social media analysis. It helps in such tasks as identifying popular words, detecting anomalies, understanding customer preferences, and making data-driven decisions.
In simpler terms, this frequency count helps to gain a deeper understanding and insights into the data we have.

What is CM Sketch?

As mentioned above CM Sketch is a probabilistic data structure, which will act as a frequency table for the stream of data.
CM Sketch consists of multiple hash functions and a 2-D array for counters.
CM Sketch offers O(1) for inserting and retrieval so that we can use it for high-speed data streams such as network monitoring, stock market analysis, traffic analysis, and data mining.

Importance of CM Sketch

Approximate Counting: CM Sketch is designed to estimate the frequency of items in a data stream with limited memory.
Memory Efficiency: The Count-Min Sketch (CM Sketch) demonstrates sub-linear memory usage, as it employs a fixed amount of memory regardless of the super-linear size of the data stream it processes.
Streaming Data Analysis: CM Sketch is specifically well-suited for processing data streams, where data arrives continuously and cannot be stored entirely due to its volume.
Scalability: As CM Sketch's memory requirements are fixed and independent of the data stream size, it offers a scalable solution for processing large and continuously growing data sets without becoming impractical as the data volume increases.

Super Linear vs Linear vs Sub Linear

In algorithmic analysis we use terms like Super Linear, Linear, Sub Linear which are used to describe the recourse consumption(time or space).

Super-linear: A function or algorithm is considered super-linear if its resource consumption grows faster than the size of the input. Like when we do recursion or nested for loops the time complexity increases at the faster rate. We can consider an algorithms which is greater than O(N) (N is size of input) will be Super Linear

Linear: A function or algorithm is linear if its resource consumption grows proportionally with the size of the input. We can consider an algorithms which is equals to O(N).

Sub-linear: A function or algorithm is considered sub-linear if its resource consumption grows at a slower rate than the size of the input. We can consider an algorithm's time complexity is O(log n). Here even if we double the input it will not double the resource allocation.

Why not Hash Tables?

When it comes to frequency count, our natural inclination is to consider Hash Tables, since hash tables are the most commonly used data structure for storing frequency counts. Hash tables could give the exact count of the data, along with this it will provide efficient insert and retrieval of data. However, there are some reasons why CM Sketch can be preferred over Hash tables for the below reasons

Efficient Memory: CM Sketch requires sufficiently less memory(Sub Linear Memory) compared to Hash Tables. We can achieve this by using a fixed 2-D array instead of storing individual k-v pairs in a table.
Streaming Data: CM Sketch is specially designed for handling streaming data, which mean element arrives one by one and may not fit in the memory. We can process the data in a single pass without storing the entire dataset. In streaming data we couldn't look back at the old data, Here in CM sketch we can perform all operations in real-time.
Scalable: CM Sketch can be easily distributed across multiple servers, Each sketch can handle a data stream independently. Also, we can merge the results to find the overall analysis. CM Sketch is well suited for large-scale distributed systems due to its memory efficiency and scalability.
Simple and faster updates: It has efficient update operations. It performs O(1) time for each element to get processed(insert or retrieval). The combination of space and time complexity gives high efficiency for the stream of data

Problems in CM Sketch

Approximation Error: Unlike Hash tables, CM Sketch will not give an exact count. CM Sketch is a probabilistic data structure that majorly focuses on memory efficiency while providing approximate data. This trade-off will be suitable in places where having an approximate frequency is enough, and converging memory is valuable.
False Positive: Count-Min Sketch may produce false positives, where the estimated frequency is higher than the actual frequency. This can occur when two different elements hash to the same counters.
Hash Functions selection: We should select our hash function to be good enough to handle more data, Poorly chosen hash functions will lead to collisions and reduced accuracy.
Set of elements: CM Sketch can process the predefined input data sets, we should know more about the data set we have before processing

Unveiling the Inner Workings

Let's explore the real backend implementations of CM Sketch.

For more accuracy of data, we can use more hash functions.

CM Sketch is a sublinear space since we have a predefined sketch/size for the counter.

High-level components needed.

Counter Array: This is a 2D array that has a fixed number of rows and columns. The Sketch in CM Sketch is nothing but a 2D array of counters
Rows: → Number of Hash Functions
Columns: → 0 to Maximum number which each hash function gives.
Hash Functions: CM Sketch uses multiple hash functions which independently determine the indices within the counter array for each element.
Incrementing Counter: When we get an element from the datastream, each hash function will process on go to calculate the respective row and column in the counter array.
Estimate Frequency: To estimate the frequency of the element, the sketch retrieves the minimum value among the counters in the row mapped by each hash function for the element. Min value gives the lower bound approximation of the element's frequency.

This special data structure is known as Count Min Sketch since we are counting the minimum value of a counter inside a sketch.

Unveiling the Inner Workings

High-Level Working Component

Consider the below data stream which has a defined set of elements.

stream_data → [A, B, K, A, A, K, S, ....∞]

First, we have to build predicted hash function outputs for the elements we have. We can use as many hash functions as possible to avoid collision.

Now initiate the CM Sketch with the Number of Rows as the length of Hash Functions and the Number of columns as the max number which hash returns. And values in the matrix/grid will be 0

In our case we use 4 hash functions out of those hash function's result max hash value is 6

Rows → 4
Columns → 7 (0-6)

CM Sketch

With 4 rows and 7 columns, we created a counter array that will store the count/frequency.

Steps
1) When we start processing the datastream first we will pass A to the CM Sketch, And for A,
→ We need to increment the counter in the indices where the hash function return value.
→ in our example A pass through hash functions H1, H2, H3, and H4 and get the values 1, 6, 3, 1.
→ So updated those values with the respective hash function's row and its return value.

2) Similarly for B,
→ We need to increment the counter in the indices where the hash function return value.
→ now, B passes through hash functions H1, H2, H3, and H4 and gets the values 1, 2, 4, 6.
→ So updated those values with the respective hash function's row and its return value.
→ Here in H1, for A and B return 1, so we need to add 1 in the index.

3) Similarly for K,
→ We need to increment the counter in the indices where the hash function return value.
→ now, K passes through hash functions H1, H2, H3, and H4 and gets the values 3, 4, 1, 6.
→ So updated those values with the respective hash function's row and its return value.

4) Now again we are getting A,
→ Now we need to add the values from the hash.
→ For A hash functions H1, H2, H3, and H4 will return 1, 6, 3, 1.
→ Now we need to add 1 with those index values already present.

5) Again we are getting A,
→ Repeat step 4.
→ For A hash functions H1, H2, H3, and H4 will return 1, 6, 3, 1.
→ Now we need to add 1 with those index values already present.

6) Now the element will be K,
→ Repeat the same step with K's hash value
→ For K hash functions H1, H2, H3, and H4 will return 3, 4, 1, 6.
→ Now we need to add 1 with those index values already present.

7) Now we are getting a new element S,
→ Repeat the same step with S's hash value
→ For S hash functions H1, H2, H3, and H4 will return 6, 2, 4, 1.
→ Now we need to add 1 with those index values already present.

Thus, we generate the CM Sketch for the data stream. We can retrieve the count at any moment we need.

How to get the frequency

Consider we need to retrieve the frequency of A.
We know that A's Hash function will return the index 1, 6, 3, 1.
Get the hash frequency from the CM Sketch Matrix for those indices.
Hash Frequency for A is,
```
A  → [4, 3, 3, 4]
```
So from the hash frequency we will be taking the minimum value as a count/frequency, which is 3 in our case
And we have 3 A's in our data stream.
Similarly, for K, we have the hash frequency.
```
K  → [2, 2, 2, 3]
```
And minimum of the frequency list are 2, and we have 2 K's in the stream.

⚠️ Inaccuracy !!

When we count minimum frequency for B
B → [4, 2, 2, 3]
min_count = 2 ≠ count of B in the stream

Considerations

We need to consider minimum value because hash collision by another hash update might hit the accuracy.
When we increase the number of hash we will get a more accurate frequency.

In Python

In Python we can use the same CM Sketch using the package/library, pyprobables

Prerequisite:

Python Version > 3.6
pyprobables package, and we can install it through pip
```
pip install pyprobables
```
We can import CountMinSketch class from pyprobables and can implement this datastructure.

Code Breakthrough

CMS = CountMinSketch(4, 1000).
This will create 4 hash functions internally and each hash function will return values in the range 0-999.
We add datastream to cms (cms.add), and cms internally do the incrementations with respect to the hash functions and elements from the data stream.
cms.check({element}) will return the count/frequency of the element/
For this input, we will get the below output.

That's how we can use CM Sketch in Python for streaming data.

Memory Comparison

Now, let try to compare the memory utilization for Hash Tables vs CM Sketch.

Consider we have a large input which will create a random word of length of 10³.

And this is the result that we can see, we have almost saved around 97% of memory with the CM sketch.

Summary

CM Sketch is a probabilistic data structure used for approximate frequency estimation of elements in a stream of data.
CM Sketch provides an efficient way to estimate frequencies by using limited memory compared to exact counting methods.
CM Sketch is a 2D matrix/grid which will increment the count with respect to the Hash Functions.
Each cell in the CM Sketch will store the frequency of the elements.

Takeaway

CM Sketch is an Approximate Frequency Estimation.
CM Sketch is highly memory efficient.
CM Sketch is highly scalable and we can merge the final output with multiple servers output.
Consider multiple hashes to improve the accuracy and reduce the collision.
Also it's major point to consider with Time vs Space. So its better to do some research before we use it in the specific use case.

Applications

Network Traffic Analyze.
Monitoring.
Data Streaming Algorithm.
Compressed Sensing.
NLP/ML

Reference

https://en.wikipedia.org/wiki/Count%E2%80%93min_sketch
Graham Cormode's CM Sketch: https://www.cs.dartmouth.edu/~ac/Teach/CS49-Fall11/Notes/lecnotes.pdf

Code Repo:

Disclaimer:

Maximizing Python Concurrency: A Comparison of Thread Pools and Threads

Ajay Balakumaran — Thu, 18 May 2023 07:44:53 +0000

Table of Content

Introduction

What is Concurrency
Why is Concurrency Important?
Problems in Concurrency.
What are Threads?
What is a Threadpool?

Thread vs. Threadpool

Definition
Usage
Pros & Cons
Difference b/w Threads and Threadpools

Implementing Threadpool in Python

Various methods available in Threadpools
- submit() Method
- map() Method
- shutdown() Method
- result() Method
Creating a Threadpool
Quick fixes
Best practices for using thread pools
When to use thread pools vs threads

Queues

What is a Queue?
Implementing a Task Management System with a Queue

Conclusion

Summary of Key Points
Takeaway

References

References

Introduction

Python is a popular programming language for building scalable and concurrent applications. One of the key features of Python is the ability to create and manage multiple threads of execution. Threads can help you take advantage of multi-core processors and speed up your program by allowing you to execute multiple tasks concurrently. However, managing threads can be complex, and creating too many threads can cause performance issues due to the overhead involved in creating and switching between threads.

Concurrency

Concurrency is a critical aspect of modern software development, allowing programs to execute multiple tasks concurrently and take advantage of multi-core processors. Python is a popular programming language that supports concurrent programming using threads. However, managing threads can be complex and can cause performance issues if not done correctly. To address some of these challenges, Python provides a mechanism for creating and managing thread pools. In this article, we'll explore the differences between thread pools and threads in Python and discuss when to use each approach to achieve better performance. By the end of this article, you'll have a clear understanding of thread pool vs threads in Python and how to choose the right approach for your specific needs.
In simpler terms, concurrency enables multiple processes or threads to execute simultaneously.

Importance of Concurrency

It can improve program performance and reduce latency by allowing multiple tasks to execute in parallel rather than sequentially.
It can make a program more responsive by allowing it to perform non-blocking operations while waiting for other tasks to complete.
It can improve resource utilization by enabling a program to make better use of available CPU and I/O resources.
It can enable more complex and sophisticated program designs by allowing for greater modularity and flexibility.
It can also be essential for certain types of programs, such as real-time systems or distributed systems, which require concurrent execution to meet their performance and scalability requirements.

Problems in Concurrency

Deadlocks and race conditions: Deadlocks and race conditions can occur when multiple threads access shared resources simultaneously. This can lead to unpredictable behavior and program crashes.
Excessive resource usage: Creating too many threads can cause excessive resource usage, leading to performance issues.
Slow performance: Concurrent programs can sometimes run slower than sequential programs due to the overhead involved in managing multiple threads.
Difficulty in debugging: Debugging concurrent programs can be challenging, as the behavior of threads can be unpredictable.
GIL limitations: The Global Interpreter Lock (GIL) in Python can limit the performance of concurrent programs that involve CPU-bound tasks.

What are threads?

A thread is a sequence of instructions that can be executed concurrently with other threads in the same program. Python threads are lightweight, meaning they require less memory and fewer resources than full-fledged processes. You can create a thread in Python by using the threading module. Here's an example of creating a thread in Python:



import threading
def my_func():
    print("Hello from thread!")

if __name__ == '__main__':
    t = threading.Thread(target=my_func)
    t.start()

In this code snippet, we define a function my_func that simply prints a message. We then create a new thread by passing this function as the target to the Thread constructor. Finally, we start the thread using the start method.

What is a thread pool?

A thread pool is a collection of threads that are created in advance and can be reused to perform tasks, rather than creating a new thread every time a task needs to be executed. Thread pools can help improve performance and reduce overhead by limiting the number of threads created and managing their lifecycle more efficiently. In Python, you can create a thread pool using the concurrent.futures module. Here's an example of using a thread pool in Python:



import concurrent.futures
def my_func():
    print("Hello from thread!")

if __name__ == '__main__':
    with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor:
        executor.submit(my_func)

Here, we define a function my_func that simply prints a message. We then create a new thread pool using the ThreadPoolExecutor class from the concurrent.futures module. We set the maximum number of workers to 2 using the max_workers argument. Finally, we submit the my_func function to the thread pool using the submit method.

Threads vs Threadpools

Definitions

Thread: A thread is the smallest unit of execution within a program. It is a lightweight process that can execute independently within the context of a larger program. Multiple threads can run concurrently within the same process, allowing for parallelism and improved performance. Threads share the same memory space and resources as the parent process, which can lead to synchronization and race condition issues.
Threadpool: A threadpool is a collection of worker threads that are created in advance and maintained by a threadpool manager. The purpose of a threadpool is to improve performance by reducing the overhead associated with creating and destroying threads. Instead of creating a new thread for each task, a threadpool can reuse existing threads, which can reduce the overhead of thread creation and destruction.

Usage

Threads are commonly used in programs that require parallelism and concurrency, such as servers, scientific simulations, and multimedia applications. They can also be used in GUI applications to improve responsiveness and performance.
Threadpools are typically used in programs that require the execution of multiple, independent tasks, such as web servers or database applications. By using a threadpool, the program can improve performance and scalability by reducing the overhead of thread creation and destruction.

Pros & Cons

Pros:
Thread pools have several advantages over threads in Python:

Resource management: Thread pools can limit the number of threads created, which can reduce resource usage and improve performance. By managing the lifecycle of threads more efficiently, thread pools can also prevent issues like thread leaks.
Scalability:: Thread pools can improve scalability by processing tasks concurrently using a fixed number of threads. This can reduce the overhead associated with creating and managing multiple threads.
Debugging: Thread pools can make debugging easier by allowing you to track the execution of tasks using a pool-specific logging mechanism. This can help you identify any issues with thread synchronization or shared resource access.

Cons:

Performance: If the number of worker threads in a thread pool is too low, performance can be slower than using threads directly. This is because thread pools may introduce overhead in managing the task queue and worker threads.
Complexity: Thread pools can be more complex to manage than threads, especially if you need to use features like timeouts, futures, or callbacks.

Differences

Creation: When using threads, each thread is created and destroyed individually as needed. When using a thread pool, a fixed number of threads are created upfront and are reused to handle different tasks.
Overhead: Creating and destroying threads can be expensive in terms of time and resources. Thread pools can reduce this overhead by reusing existing threads, leading to better performance in some cases.
Scalability: Thread pools can be more scalable than using threads individually, especially for I/O-bound tasks. This is because thread pools can manage the number of threads more effectively, avoiding situations where there are too many threads competing for system resources.
Control: When using threads, it's up to the programmer to manage the creation, destruction, and synchronization of threads. When using a thread pool, this management is handled by the pool itself, allowing the programmer to focus on defining the tasks to be executed.
Heterogeneous vs Homogeneous Tasks: The ThreadPool is for heterogeneous tasks, whereas Thread is for homogeneous tasks.
Reuse vs Single-Use: The ThreadPool supports reuse, whereas the Thread class is for single use.

Implementing Threadpool in Python

Various methods available in Threadpools

submit(): Submits a task to the thread pool for execution.
map(): Applies a function to a list of arguments, submitting each to the thread pool for execution.
shutdown(): Waits for all submitted tasks to complete before returning.
result(): Returns the result of a completed task.

Creating a Threadpool



import concurrent.futures
#Define a function that will be executed in the thread pool
def my_function(arg):
    #Perform some long-running operation here
    result = arg * 2
    return result
#Create a thread pool object with 5 worker threads
with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
    #Submit tasks to the thread pool using the submit() method
    future1 = executor.submit(my_function, 1)
    future2 = executor.submit(my_function, 2)
    future3 = executor.submit(my_function, 3)

    #Use the map() method to apply a function to a list of arguments
    queue = [4, 5, 6]
    results = executor.map(my_function, queue)

    #Wait for all tasks to complete using the shutdown() method
    executor.shutdown()

    #Get the results of each task using the result() method
    result1 = future1.result()
    result2 = future2.result()
    result3 = future3.result()
    #Print the results
    print(result1, result2, result3)
    print(list(results))

In this sample implementation, we first define a simple function called my_function() that takes a single argument and performs a long-running operation on it. We then create a thread pool object with a maximum of 5 worker threads, and submit three tasks to it using the submit() method. We also use the map() method to apply the same function to a list of arguments.

After submitting all the tasks, we wait for all of them to complete using the shutdown() method. We then use the result() method to retrieve the results of each individual task that we submitted earlier. Finally, we print out the results of each task as well as the results of the map() operation.

Note: that in this implementatuon, we are using the submit() method to submit individual tasks, and the map() method to apply a function to a list of arguments. After that we use the shutdown() method to wait for all tasks to complete before retrieving the results using the result() method.

Quick fixes using Threadpools

1) Deadlocks and race conditions: Thread pools can help avoid deadlocks and race conditions by managing the number of threads accessing shared resources. A thread pool maintains a queue of tasks and assigns them to a fixed number of worker threads. This ensures that only a limited number of threads access shared resources at any given time.

2) Excessive resource usage: Thread pools can limit the number of threads created, which can reduce resource usage and improve performance. By managing the lifecycle of threads more efficiently, thread pools can also prevent issues like thread leaks.

3) Slow performance: Thread pools can improve performance by batching tasks and executing them concurrently using a fixed number of threads. This can reduce the overhead associated with creating and managing multiple threads.

4) Difficulty in debugging: Thread pools can make debugging easier by allowing you to track the execution of tasks using a pool-specific logging mechanism. This can help you identify any issues with thread synchronization or shared resource access.

5) GIL limitations: Thread pools can help avoid GIL limitations by using multiple processes instead of threads. The multiprocessing module in Python provides a pool of worker processes that can execute tasks in parallel without being constrained by the GIL.

In summary, thread pools can provide a more efficient and manageable approach to concurrency in Python by avoiding some of the common problems associated with threads. By using thread pools, you can write concurrent programs that are scalable, robust, and performant.

Best practices for using thread pools

Here are some best practices to keep in mind when using thread pools in Python:

Set the right number of worker threads: The optimal number of worker threads depends on the number of available CPU cores and the nature of the tasks being executed. Experiment with different values to find the optimal configuration for your application.
Use a bounded task queue: To prevent the task queue from growing too large, you can use a bounded queue (e.g., queue.Queue(maxsize=10)) that raises an exception when the queue is full.
Handle exceptions: Make sure to handle exceptions that occur in worker threads. Otherwise, exceptions can go unnoticed and cause hard-to-debug errors.
Use Context Managers: To ensure that thread pools are properly cleaned up when they're no longer needed, use a context manager (with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor) to automatically shut down the pool when the block is exited.
Avoid shared state: If possible, avoid sharing state between worker threads to prevent race conditions and other synchronization issues.

By following these best practices, we can ensure that our Python application is using thread pools efficiently and effectively.

When to use Threadpool vs Thread

The decision to use thread pools or threads in Python depends on the specific requirements and various factors of your application. Here are some general guidelines:

I/O-bound tasks: If your application performs many I/O-bound tasks (e.g., reading from a database or making API requests), using thread pools can be beneficial. I/O-bound tasks spend a significant portion of their time waiting for external resources, such as network responses or disk I/O. By utilizing thread pools, multiple workers can be assigned to handle these tasks concurrently, allowing other workers to continue executing while waiting for I/O operations to complete. This can improve overall performance and responsiveness.
CPU-bound tasks: If your application performs many CPU-bound tasks (e.g., image processing or machine learning), utilizing multithreading can be considered. While the Global Interpreter Lock (GIL) in Python prevents true parallelism, using multithreading can still be beneficial in certain cases. CPU-bound tasks utilize system resources extensively, and distributing the workload across multiple threads can maximize CPU utilization. However, it's important to note that due to the GIL, the performance gains achieved through multithreading for CPU-bound tasks may be limited compared to other approaches like multiprocessing or native extensions that release the GIL. It's advisable to evaluate the specific requirements of your application and consider alternative approaches depending on the nature of the task and available resources.
Hybrid tasks: If your application performs a mix of I/O-bound and CPU-bound tasks, you can experiment with thread pools and threads to find the best performance.

Queue

What is queue?
A queue is a collection of elements that follow the "first-in, first-out" (FIFO) principle. In Python, a queue can be implemented using the queue module. Queues are often used in concurrent programming to manage tasks or messages that need to be processed in a specific order

Implementing a Task Management System with a Queue

Create a Queue object: To use a queue in Python, you can create a Queue object from the queue module. The Queue class provides methods to add and remove elements from the queue.
Define tasks as functions: In a task management system, each task is a function that performs a specific action. When a task is added to the queue, it is added as a function object.
Add tasks to the queue: To add a task to the queue, you can use the put method of the Queue object. The put method adds the task to the end of the queue.
Start worker threads: To process tasks in the queue, you need to create one or more worker threads. A worker thread is a separate thread of execution that pulls tasks from the queue and processes them. In Python, you can create a worker thread by defining a function that runs in an infinite loop and calls the get method of the Queue object to retrieve tasks.
Retrieve tasks from the queue: To retrieve tasks from the queue, you can use the get method of the Queue object. The get method removes the first element from the queue and returns it.
Process tasks: Once a worker thread retrieves a task from the queue, it can execute the task by calling the function object with the appropriate arguments.

Conclusion:

Summary Key Points

Concurrency is essential for improving the performance of Python programs.
Threads and threadpools are popular concurrency models in Python, with each having its own strengths and weaknesses.
Threadpools offer several advantages over threads, such as efficient resource utilization and better error handling.
However, threads are still useful in certain scenarios, such as when dealing with short-lived tasks or when the number of tasks is limited.
A task management system with a queue is another way to handle concurrency, particularly when dealing with a large number of tasks.
Queues ensure that tasks are executed in a first-in, first-out order and help manage the workload of the program.
When choosing between threads, threadpools, and queues, it's important to consider the specific requirements of the program.
To improve the performance and efficiency of Python programs, it's crucial to understand the different concurrency models and when to use them.

Takeaway

1) Understand the basic concepts of threads and thread pools in Python.
2) Identify scenarios where thread pools or threads might be the better approach for their specific requirements.
3) Learn how to implement threads and thread pools in Python, and how to handle common issues such as synchronization and error handling.
4) Understand the impact of threads and thread pools on system resources and performance, and how to optimize their usage for maximum efficiency.
5) Have a clear idea of how to choose between threads and thread pools for their own Python projects.

Reference

Python's concurrent.futures documentation: https://docs.python.org/3/library/concurrent.futures.html
Python's threading documentation: https://docs.python.org/3/library/threading.html
Real Python's guide to concurrency in Python: https://realpython.com/python-concurrency/

Disclaimer:

Demystifying Azure Form Recognizer

Ajay Balakumaran — Tue, 15 Nov 2022 10:32:37 +0000

Introduction

Every industry has made a significant transformation in the recent decade. And it’s looking for even more efficient and optimized work. We live in a generation where data plays a major role in various domains, and everyone is aware of the importance of data.
Organizations will receive or collect data in a various way. But they need to keep all the information in one place to access it efficiently and swiftly. Most sectors need to extract the data from the source they receive. And they know it would be a repetitive task and require more effort every time.
Therefore, organizations need to find a solution that reduces human error in extractions and increases efficiency in extracting the data from the input sources.

Quick Fix:

To solve the above problems in data extraction from structured and semi-structured documents automatically, we can use Azure Form Recognizer.

Form Recognizer:

Form recognizer is a cognitive service that uses Machine Learning technology to identify and extract the required data.

Form Recognizer uses deep learning models and enables us to train with custom sample models to fetch the details we required.

Get Started:

Prerequisites:

To get started with the Form Recognizers, we will need the below.
Python 3.7 or later is required to use this package.
You must have an Azure subscription and a Cognitive Services or Form Recognizer resource to use this package.
Azure Form Recognizer client library for Python.

Create Form Recognizers:

Select the Subscriptions and Resources Group (create one)
Select region as the closet region.
Provide a name for the Form Recognizer.
Select pricing tier.
Review and create it.

Get Keys and Endpoints:

Open the created form recognizers.
Check for keys and endpoints on the right side.

Get ready with the input files:

Sample documents which we used for this demo

Python SDK:

Create a virtual environment and install the Azure module.



pip install azure-ai-formrecognizer

Save the keys and endpoint in a config file to call the API from Python.
Install required modules and libraries and import.



from azure.ai.formrecognizer import DocumentAnalysisClient
from azure.core.credentials import AzureKeyCredential

config = json.load(open('config.json'))
api_key = config['APIKEY']
endpoint = config['APIENDPOINT']
credential = AzureKeyCredential(api_key)

Once we get the input file, we have to parse the input file and pass it to the API along with the credentials.



with open("input_files/test_file.pdf", "rb") as fd:
    document = fd.read()

poller = document_analysis_client.begin_analyze_document("prebuilt-layout", document)
result = poller.result()

We will be receiving the result in azure.ai.formrecognizer type.
We can parse over the result and get the data we need ```python

for table_idx, table in enumerate(result.tables):
print(
"Table # {} has {} rows and {} columns".format(
table_idx, table.row_count, table.column_count
)
)
for region in table.bounding_regions:
print(
"Table # {} location on page: {} is {}".format(
table_idx,
region.page_number,
region.polygon
)
)
for cell in table.cells:
print(
"...Cell[{}][{}] has content '{}'".format(
cell.row_index,
cell.column_index,
cell.content,
)
)




  This will give the data of each and every cell in the table




  
  
  Other Functions:




for page in result.pages:
    # prints data in each and every pages in the file
    # prints page height, width
    # syntax - page.width, page.height, page.unit
    pass
    for line in page.line:
        # prints each and every lines in a page        
        # syntax - line.context
        pass

    for word in page.word:
        # print all words along with its positions and confidence score
        # syntax - word.context, word.confidence, word.position
        pass


</code></pre></div><h2>
  <a name="key-value-pair-extraction" href="#key-value-pair-extraction">
  </a>
  Key Value Pair Extraction:
</h2>

<ul>
<li>  Similar to table extraction Form Recognizer will help in extracting the key value pair data from document as shown below</li>
</ul>

<p><img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/n8fsqnx88qv72rw1zxuy.png" alt="Image description"></p>

<p>Image from <a href="https://azure.microsoft.com/en-us/products/cognitive-services/#features">Microsoft Azure Cognitive Services Demos</a></p>
<h2>
  <a name="different-models-available" href="#different-models-available">
  </a>
  Different Models available:
</h2>

<table>
    <thead>
        <tr>
            <th>Model Type</th>
            <th>Model Name</th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td rowspan="3">Document analysis model</td>
            <td><a href="https://learn.microsoft.com/en-us/azure/applied-ai-services/form-recognizer/concept-read?view=form-recog-3.0.0">Read OCR model</a></td>
        </tr>
        <tr>
            <td><a href="https://learn.microsoft.com/en-us/azure/applied-ai-services/form-recognizer/concept-general-document?view=form-recog-3.0.0">General document model</a></td>
        </tr>
        <tr>
            <td><a href="https://learn.microsoft.com/en-us/azure/applied-ai-services/form-recognizer/concept-layout?view=form-recog-3.0.0">Layout analysis model</a></td>
        </tr>
    </tbody>
    <tbody>
        <tr>
            <td rowspan="5">Prebuild Models</td>
            <td><a href="https://learn.microsoft.com/en-us/azure/applied-ai-services/form-recognizer/concept-w2?view=form-recog-3.0.0">W-2 form model</a></td>
        </tr>
        <tr>
            <td><a href="https://learn.microsoft.com/en-us/azure/applied-ai-services/form-recognizer/concept-invoice?view=form-recog-3.0.0">Invoice model</a></td>
        </tr>
        <tr>            
            <td><a href="https://learn.microsoft.com/en-us/azure/applied-ai-services/form-recognizer/concept-receipt?view=form-recog-3.0.0">Receipt model</a></td>
        </tr>
        <tr>
            <td><a href="https://learn.microsoft.com/en-us/azure/applied-ai-services/form-recognizer/concept-id-document?view=form-recog-3.0.0">Identity document model</a></td>
        </tr>
        <tr>
            <td><a href="https://learn.microsoft.com/en-us/azure/applied-ai-services/form-recognizer/concept-business-card?view=form-recog-3.0.0">Business card model</a></td>
        </tr>
    </tbody>
    <tbody>
        <tr>
            <td rowspan="2">Custom Models</td>
            <td><a href="https://learn.microsoft.com/en-us/azure/applied-ai-services/form-recognizer/concept-custom?view=form-recog-3.0.0">Custom Model</a></td>
        </tr>
        <tr>
            <td><a href="https://learn.microsoft.com/en-us/azure/applied-ai-services/form-recognizer/concept-model-overview?view=form-recog-3.0.0">Composed Model</a></td>
        </tr>
    </tbody>
</table>
<h2>
  <a name="how-to-select-a-model" href="#how-to-select-a-model">
  </a>
  How to select a model:
</h2>

<table>
    <thead>
        <tr>
            <th>Document type</th>
            <th>Data to Extract</th>
            <th>Best Model</th>
        </tr>
    </thead>
    <thead>
        <tr>
            <td>A generic document like a contract or letter.</td>
            <td>You want to extract primarily text lines, words, locations, and detected languages.</td>
            <td>Read OCR model</td>
        </tr>  
    </thead>
    <thead>
        <tr>
            <td>A document that includes structural information like a report or study.</td>
            <td>In addition to text, you need to extract structural information like tables, selection marks, paragraphs, titles, headings, and subheadings.</td>
            <td>Layout analysis model</td>
        </tr>  
    </thead>
    <thead>
        <tr>
            <td>A structured or semi-structured document that includes content formatted as fields and values, like a credit application or survey form.</td>
            <td>You want to extract fields and values including ones not covered by the scenario-specific prebuilt models without having to train a custom model.</td>
            <td>General document model</td>
        </tr>  
    </thead>
    <thead>
        <tr>
            <td>U.S. W-2 form</td>
            <td>You want to extract key information such as salary, wages, and taxes withheld from US W2 tax forms.</td>
            <td>W-2 model</td>
        </tr>  
    </thead>
    <thead>
        <tr>
            <td>Invoice</td>
            <td>You want to extract key information such as customer name, billing address, and amount due from invoices.</td>
            <td>Invoice model</td>
        </tr>  
    </thead>
    <thead>
        <tr>
            <td>Receipt</td>
            <td>You want to extract key information such as merchant name, transaction date, and transaction total from a sales or single-page hotel receipt.</td>
            <td>Receipt model</td>
        </tr>  
    </thead>
    <thead>
        <tr>
            <td>Identity document (ID) like a passport or driver's license.</td>
            <td>You want to extract key information such as first name, last name, and date of birth from US drivers' licenses or international passports.</td>
            <td>Identity document (ID) model</td>
        </tr>  
    </thead>
    <thead>
        <tr>
            <td>Business card</td>
            <td>You want to extract key information such as first name, last name, company name, email address, and phone number from business cards.</td>
            <td>Business card model</td>
        </tr>  
    </thead>
    <thead>
        <tr>
            <td>Mixed-type document(s)</td>
            <td>You want to extract key-value pairs, selection marks, tables, signature fields, and selected regions not extracted by prebuilt or general document models.</td>
            <td>Custom model</td>
        </tr>  
    </thead>
</table>
<h2>
  <a name="filedocument-formats" href="#filedocument-formats">
  </a>
  File/Document formats:
</h2>

<p>PDF, Images, TIFF files can be used in the Form Recognizer.</p>
<h2>
  <a name="limitations" href="#limitations">
  </a>
  Limitations:
</h2>

<ul>
<li>  Form Recognizer doesn’t have a pre-build model for generic form extraction, if we need to get the form data from a document which is not in English we need to train the model</li>
<li>  Total size of training data set must be less than 500 pages</li>
<li>  We can pass a file(PDF, TIFF) of size upto 500MB and 2000 pages.</li>
<li>  Page dimensions can be upto 10k x 10k pixles for Images and 17 x 17 for PDFs</li>
<li>      Extraction may fail if the table contains only one column.</li>
</ul>
<h2>
  <a name="programming-languages" href="#programming-languages">
  </a>
  Programming Languages:
</h2>

<p>Form recognizers supports below programming languages with the SDK and libraries</p>

<ul>
<li>  Python </li>
<li>  Java</li>
<li>  C# </li>
<li>  Javascript</li>
</ul>
<h2>
  <a name="conclusion" href="#conclusion">
  </a>
  Conclusion:
</h2>

<p>This data extractions process can be streamlined with the help of the inclusion of AI/ML in organizations will help in reduce error and increase the efficiency of work.<br>
For a larger organization, it could be a quick process with a more accurate level of extraction with high efficiency helps to focus on the next level in pipelines.<br>
Along with Azure Form Recognizers, we have other services/tools like Instabase, AWS Textract are also highly effective tools which is available in the market.</p>
<h2>
  <a name="references" href="#references">
  </a>
  References:
</h2>

<ul>
<li><a href="https://learn.microsoft.com/en-us/azure/?view=form-recog-3.0.0&amp;product=popular">https://learn.microsoft.com/en-us/azure/?view=form-recog-3.0.0&amp;product=popular</a></li>
<li><a href="https://www.maxinai.com/blog/2021/03/12/ai-document-data-extraction-financial-institutions/">https://www.maxinai.com/blog/2021/03/12/ai-document-data-extraction-financial-institutions/</a></li>
<li><a href="https://azure.microsoft.com/en-us/products/cognitive-services/#features">https://azure.microsoft.com/en-us/products/cognitive-services/#features</a></li>
</ul>
<h2>
  <a name="disclaimer" href="#disclaimer">
  </a>
  Disclaimer:
</h2>

<p>This is a personal blog. The views and opinions expressed here are only those of the author and do not represent those of any organization or any individual with whom the author may be associated, professionally or personally.</p>