DEV Community: Daniel Jarvis

[Python-CV2] Image Segmentation : Canny Edges, Watershed, and K-Means Methods

Daniel Jarvis — Tue, 10 Dec 2024 07:47:15 +0000

Segmentation is a fundamental technique in image analysis that allows us to divide an image into meaningful parts based on objects, shapes, or colors. It plays a pivotal role in applications such as object detection, computer vision, and even artistic image manipulation. But how can we achieve segmentation effectively? Fortunately, OpenCV (cv2) provides several user-friendly and powerful methods for segmentation.

In this tutorial, we’ll explore three popular segmentation techniques:

Canny Edge Detection – perfect for outlining objects.
Watershed Algorithm – great for separating overlapping regions.
K-Means Color Segmentation – ideal for clustering similar colors in an image.

To make this tutorial engaging and practical, we’ll use satellite and aerial images from Osaka, Japan, focusing on the ancient kofun burial mounds. You can download these images and the corresponding sample notebook from the tutorial's GitHub page.

Canny Edge Detection to Contour Segmentation

Canny Edge Detection is a straightforward yet powerful method to identify edges in an image. It works by detecting areas of rapid intensity change, which are often boundaries of objects. This technique generates "thin edge" outlines by applying intensity thresholds. Let’s dive into its implementation using OpenCV.

Example: Detecting Edges in a Satellite Image
Here, we use a satellite image of Osaka, specifically a kofun burial mound, as a test case.

import cv2 
import numpy as np
import matplotlib.pyplot as plt
files = sorted(glob("SAT*.png")) #Get png files 
print(len(files))
img=cv2.imread(files[0])
use_image= img[0:600,700:1300]
gray = cv2.cvtColor(use_image, cv2.COLOR_BGR2GRAY)

#Stadard values 
min_val = 100
max_val = 200
# Apply Canny Edge Detection
edges = cv2.Canny(gray, min_val, max_val)
#edges = cv2.Canny(gray, min_val, max_val,apertureSize=5,L2gradient = True )
False
# Show the result
plt.figure(figsize=(15, 5))
plt.subplot(131), plt.imshow(cv2.cvtColor(use_image, cv2.COLOR_BGR2RGB))
plt.title('Original Image'), plt.axis('off')
plt.subplot(132), plt.imshow(gray, cmap='gray')
plt.title('Grayscale Image'), plt.axis('off')
plt.subplot(133), plt.imshow(edges, cmap='gray')
plt.title('Canny Edges'), plt.axis('off')
plt.show()

The output edges clearly outline parts of the kofun burial mound and other regions of interest. However, some areas are missed due to the sharp thresholding. The results depend heavily on the choice of min_val and max_val, as well as the image quality.

To enhance edge detection, we can preprocess the image to spread out pixel intensities and reduce noise. This can be achieved using histogram equalization (cv2.equalizeHist()) and Gaussian blurring (cv2.GaussianBlur()).


use_image= img[0:600,700:1300]
gray = cv2.cvtColor(use_image, cv2.COLOR_BGR2GRAY)
gray_og = gray.copy()
gray = cv2.equalizeHist(gray)
gray = cv2.GaussianBlur(gray, (9, 9),1)

plt.figure(figsize=(15, 5))
plt.subplot(121), plt.imshow(gray, cmap='gray')
plt.title('Grayscale Image')
plt.subplot(122)
_= plt.hist(gray.ravel(), 256, [0,256],label="Equalized") 
_ = plt.hist(gray_og.ravel(), 256, [0,256],label="Original",histtype='step')
plt.legend()
plt.title('Grayscale Histogram')

This preprocessing evens out the intensity distribution and smooths the image, which helps the Canny Edge Detection algorithm capture more meaningful edges.

Edges are useful, but they only indicate boundaries. To segment enclosed areas, we convert edges into contours and visualize them.

# Edges to contours 
contours, hierarchy = cv2.findContours(edges, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)
# Calculate contour areas
areas = [cv2.contourArea(contour) for contour in contours]

# Normalize areas for the colormap
normalized_areas = np.array(areas)
if normalized_areas.max() > 0:
    normalized_areas = normalized_areas / normalized_areas.max()

# Create a colormap
cmap = plt.cm.jet

# Plot the contours with the color map
plt.figure(figsize=(10, 10))
plt.subplot(1,2,1)
plt.imshow(gray, cmap='gray', alpha=0.5)  # Display the grayscale image in the background
mask = np.zeros_like(use_image)
for contour, norm_area in zip(contours, normalized_areas):
    color = cmap(norm_area)  # Map the normalized area to a color
    color = [int(c*255) for c in color[:3]]
    cv2.drawContours(mask, [contour], -1, color,-1 )  # Draw contours on the image

plt.subplot(1,2,2)

The above method highlights detected contours with colors representing their relative areas. This visualization helps verify if the contours form closed bodies or merely lines. However, in this example, many contours remain unclosed polygons. Further preprocessing or parameter tuning could address these limitations.

By combining preprocessing and contour analysis, Canny Edge Detection becomes a powerful tool for identifying object boundaries in images. However, it works best when objects are well-defined and noise is minimal. Next, we’ll explore K-Means Clustering to segment the image by color, offering a different perspective on the same data.

KMean Clustering

K-Means Clustering is a popular method in data science for grouping similar items into clusters, and it’s particularly effective for segmenting images based on color similarity. OpenCV's cv2.kmeans function simplifies this process, making it accessible for tasks like object segmentation, background removal, or visual analysis.

In this section, we will use K-Means Clustering to segment the kofun burial mound image into regions of similar colors.

To start, we apply K-Means Clustering on the RGB values of the image, treating each pixel as a data point.

# Kmean color segmentation
use_image= img[0:600,700:1300]
#use_image = cv2.medianBlur(use_image, 15)


 # Reshape image for k-means
pixel_values = use_image.reshape((-1, 3)) if len(use_image.shape) == 3 else use_image.reshape((-1, 1))
pixel_values = np.float32(pixel_values)

criteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 10, 1.0)
K = 3
attempts=10
ret,label,center=cv2.kmeans(pixel_values,K,None,criteria,attempts,cv2.KMEANS_PP_CENTERS)

centers = np.uint8(center)
segmented_data = centers[label.flatten()]
segmented_image = segmented_data.reshape(use_image.shape)

plt.figure(figsize=(10, 6))
plt.subplot(1,2,1),plt.imshow(use_image[:,:,::-1])
plt.title("RGB View")
plt.subplot(1,2,2),plt.imshow(segmented_image[:,:,[2,1,0]])
plt.title(f"Kmean Segmented Image K={K}")

In the segmented image, the burial mound and surrounding regions are clustered into distinct colors. However, noise and small variations in color lead to fragmented clusters, which can make interpretation challenging.

To reduce noise and create smoother clusters, we can apply a median blur before running K-Means.

# Apply median blur to smooth the image
use_image_blurred = cv2.medianBlur(use_image, 15)

The blurred image results in smoother clusters, reducing noise and making the segmented regions more visually cohesive.

To better understand the segmentation results, we can create a color map of the unique cluster colors, using matplotlib plt.fill_between;


# View colors of segmented image 
colors=np.unique(segmented_image[:,:,::-1].reshape(-1,3),axis=0)
plt.figure(figsize=(10, 2))
for i, color in enumerate(colors):
    plt.fill_between([i, i+1], 0, 1, color=color/255.0)  # Normalize RGB to [0,1] range
    plt.text(i+0.5, 0.45, f" {color}", ha='center', va='bottom')
plt.axis('off')

This visualization provides insight into the dominant colors in the image and their corresponding RGB values, which can be useful for further analysis. AS we could mask and select area my color code now.

The number of clusters (K) significantly impacts the results. Increasing K creates more detailed segmentation, while lower values produce broader groupings. To experiment, we can iterate over multiple K values.

ks = [2,3,5,7,9,12,15]
results = []
plt.figure(figsize=(15, 6))

pixel_values = use_image.reshape((-1, 3)) if len(use_image.shape) == 3 else use_image.reshape((-1, 1))
pixel_values = np.float32(pixel_values)
criteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 10, 1.0)

attempts=10

for i,k in enumerate(ks):
    print(k)
    ret,label,center=cv2.kmeans(pixel_values.copy(),k,None,criteria,attempts,cv2.KMEANS_PP_CENTERS)
    segmented_data = center[label.flatten()]
    segmented_image = segmented_data.reshape(use_image.shape)
    results.append(segmented_image)
    segmented_image=np.array(segmented_image,dtype=np.uint8)
    plt.subplot(1,len(ks),i+1)
    plt.imshow(segmented_image[:,:,[2,1,0]])
    plt.title(f"Kmean K={k}")
    plt.axis('off')

The clustering results for different values of K reveal a trade-off between detail and simplicity:

・Lower K values (e.g., 2-3): Broad clusters with clear distinctions, suitable for high-level segmentation.
・Higher K values (e.g., 12-15): More detailed segmentation, but at the cost of increased complexity and potential over-segmentation.

K-Means Clustering is a powerful technique for segmenting images based on color similarity. With the right preprocessing steps, it produces clear and meaningful regions. However, its performance depends on the choice of K, the quality of the input image, and the preprocessing applied. Next, we’ll explore the Watershed Algorithm, which uses topographical features to achieve precise segmentation of objects and regions.

Watershed Segmentation

The Watershed Algorithm is inspired by topographic maps, where watersheds divide drainage basins. This method treats grayscale intensity values as elevation, effectively creating "peaks" and "valleys." By identifying regions of interest, the algorithm can segment objects with precise boundaries. It’s particularly useful for separating overlapping objects, making it a great choice for complex scenarios like cell segmentation, object detection, and distinguishing densely packed features.

The first step is preprocessing the image to enhance the features, followed by applying the Watershed Algorithm.

# Read and preprocess the image
img=cv2.imread(files[0])
use_image= img[0:600,700:1300].copy()
# Convert to grayscale and apply threshold for enhanced feature detection 
gray = cv2.cvtColor(use_image, cv2.COLOR_BGR2GRAY)
# binary image inversion, this  ensures that foreground objects are white, and the background is black.
_, thresh = cv2.threshold(gray, 40, 255, cv2.THRESH_BINARY_INV  | cv2.THRESH_OTSU)


# Noise removal using morphological opening, this reduced segmentation artifacts 
kernel = np.ones((3, 3), np.uint8)
opening = cv2.morphologyEx(thresh, cv2.MORPH_OPEN, kernel, iterations=2)

# Distance transform to highlight object centers, this indicated distance from nearest pixel boundary 
dist_transform = cv2.distanceTransform(opening, cv2.DIST_L2, 5)
#  Thresholding identifies "sure foreground" regions, what is the areas most likely part of objects.
_, sure_fg = cv2.threshold(dist_transform, 0.7 * dist_transform.max(), 255, 0)
sure_fg = np.uint8(sure_fg)
# Identify the sure background and unknown regions
sure_bg = cv2.dilate(opening, kernel, iterations=3)
unknown = cv2.subtract(sure_bg, sure_fg)

# Create markers for watershed segmentation
markers = cv2.connectedComponents(sure_fg.astype(np.uint8))[1]
markers = markers + 1
markers[unknown == 255] = 0

# Apply Watershed, Pixels marked -1 represent boundary lines, which are visually highlighted.
markers = cv2.watershed(use_image, markers)
use_image[markers == -1] = [255, 255, 255]  # Mark boundaries in red

The segmented regions and boundaries can be visualized alongside intermediate processing steps.

#ploting process
plt.figure(figsize=(15, 6))
plt.subplot(151),plt.imshow(dist_transform)
plt.title('Distance Transform'),plt.axis('off')
plt.subplot(152),plt.imshow(sure_fg)
plt.title('Sure Foreground'),plt.axis('off')
plt.subplot(153),plt.imshow(sure_bg)
plt.title('Sure Background'),plt.axis('off')
plt.subplot(154),plt.imshow(markers)
plt.title('Watershed Segmentation'),plt.axis('off')
plt.subplot(155),plt.imshow(cv2.cvtColor(use_image, cv2.COLOR_BGR2RGB))
plt.title('Watershed Segmentation'),plt.axis('off')
plt.show()

The algorithm successfully identifies distinct regions and draws clear boundaries around objects. In this example, the kofun burial mound is segmented accurately. However, the algorithm’s performance depends heavily on preprocessing steps like thresholding, noise removal, and morphological operations.

Adding advanced preprocessing, such as histogram equalization or adaptive blurring, can further enhance the results. For instance:


# Enhanced preprocessing
gray = cv2.equalizeHist(gray)
gray = cv2.GaussianBlur(gray, (9, 9), 1)

# Repeat the thresholding and watershed steps as before
_, thresh = cv2.threshold(gray, 40, 255, cv2.THRESH_BINARY_INV | cv2.THRESH_OTSU)

With these adjustments, more regions can be accurately segmented, and noise artifacts can be minimized.

The Watershed Algorithm excels in scenarios requiring precise boundary delineation and separation of overlapping objects. By leveraging preprocessing techniques, it can handle even complex images like the kofun burial mound region effectively. However, its success depends on careful parameter tuning and preprocessing.

Conclusion

Segmentation is an essential tool in image analysis, providing a pathway to isolate and understand distinct elements within an image. This tutorial demonstrated three powerful segmentation techniques—Canny Edge Detection, K-Means Clustering, and Watershed Algorithm—each tailored for specific applications. From outlining the ancient kofun burial mounds in Osaka to clustering urban landscapes and separating distinct regions, these methods highlight the versatility of OpenCV in tackling real-world challenges.

Now go and apply some of there method to an application of your choose and comment and share the results. Also if you know any other simple segmentation methods please share too

[CV2] Motion Detection and Tracking in OpenCV: Frame Delta, MOG2, and Optical Flow Explained

Daniel Jarvis — Sun, 01 Dec 2024 11:37:47 +0000

In the previous blog posts, we explored basic image processing techniques in OpenCV.

While OpenCV is excellent for image manipulation, it also excels at processing video frames. By using cap = cv2.VideoCapture(video_file), we can analyze video streams frame by frame.

In this post, I will demonstrate motion detection and tracking techniques using a sample video of Shibuya Crossing, extracted from a public live stream. If you'd like to follow along with the same video, you can download it here: [Video Download Link].

Setting Up and Displaying the Video

First, let's load the video and retrieve its metadata. These properties—frame size, frame count, and frames per second (FPS)—are critical for accurate processing and saving of video data.

import cv2 

video_file = "./shibuy_test_20240705_063826.avi"
cap = cv2.VideoCapture(video_file)

# Test frame properties
frame_count = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
frame_width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
frame_height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
fps = cap.get(cv2.CAP_PROP_FPS)

print(f"Total Frames: {frame_count}, Width: {frame_width}, Height: {frame_height}, FPS: {fps}")

Its typical practice to set the video as a cap variable, there we can then get the meta data of the video with cv2.CAP_PROP_*. this is useful to get the frame size, total frame count and fps of the video.

Displaying the Video

To display the video, we use a while loop to read frames sequentially and show them with cv2.imshow. Always include a quit option for smooth termination.

while cap.isOpened():
    ret, frame = cap.read()
    if ret:
        cv2.imshow('Video Playback', frame)
        # Press "q" to quit
        if cv2.waitKey(1) & 0xFF == ord('q'):
            break
    else:
        break

cap.release()
cv2.destroyAllWindows()

Motion Detection

Method 1 : Frame Delta

Motion detection can be as simple as comparing consecutive frames. Using cv2.absdiff(frame_1, frame_2), we compute the absolute difference between two frames.

frame_1 = cap.read()[1]
cap.set(cv2.CAP_PROP_POS_FRAMES, 5)  # Skip to the 5th frame
frame_2 = cap.read()[1]

delta_frame = cv2.absdiff(frame_1, frame_2)
cv2.imshow('Delta Frame', delta_frame)
cv2.waitKey(0)
cv2.destroyAllWindows()

To apply this method to the entire video:

prev_frame = cap.read()[1]
while cap.isOpened():
    ret, frame = cap.read()
    if ret:
        delta_frame = cv2.absdiff(prev_frame, frame)
        cv2.imshow('Delta Frame', delta_frame)
        prev_frame = frame
        if cv2.waitKey(1) & 0xFF == ord('q'):
            break
    else:
        break

cap.release()
cv2.destroyAllWindows()

Saving the video

This above code displayed the video but it closed after the code was finished. To save this video we can use cv2.VideoWriter(). This is quite simple all we need to do is set the recorder variable then write each frame to it in the loop, then close it when done.

rec = cv2.VideoWriter('./output_delta.mp4',cv2.VideoWriter_fourcc(*'mp4v'),fps,(frame_width,frame_height),True)


#Video test 1
prev_frame=cap.read()[1]
while(cap.isOpened()):
    ret, frame = cap.read()
    if ret == True:
        delta_frame = cv2.absdiff(prev_frame,frame)
        cv2.imshow('Delta Frame', delta_frame)
        prev_frame=frame
        rec.write(delta_frame)
        # Press "q" to quit option
        if cv2.waitKey(1) & 0xFF == ord('q'):
            break
    else:
        break


cap.release()
cv2.destroyAllWindows()
rec.release()

Note is good process to just use the fps,frame_width,frame_height, from the cap values to not lead to any issues with the size of the final video.
Another option is change the output format. Its currently set to mp4 by the cv2.VideoWriter_fourcc(*'mp4v') but over formats like mov and avi can be used too.Finnaly is worth noting the True at the end, thats a default setting is_color i.e is a 3 channal image. If for example you have a 1 channal gray image this should be set to False for else it wont save properly and you get a "1kb" file of nothing at the end (Trust me this happens a lot).

Method 2: Background Subtraction

Another option is use cv2.createBackgroundSubtractorMOG2, we can separate moving objects from the static background. This a allows a indication of motion in a video and can separate the motion into objects and shadows.

fgbg = cv2.createBackgroundSubtractorMOG2(history=500, varThreshold=16, detectShadows=True)

while cap.isOpened():
    ret, frame = cap.read()
    if ret:
        fgmask = fgbg.apply(frame)
        cv2.imshow('Background Subtraction', fgmask)
        if cv2.waitKey(1) & 0xFF == ord('q'):
            break
    else:
        break

cap.release()
cv2.destroyAllWindows()

cv2.createBackgroundSubtractorMOG2(history = 500,varThreshold = 16, detectShadows = true)

history: The number of frames used to build the background model. The default value is 500.
・varThreshold: The threshold on the squared Mahalanobis distance between the pixel and the model to decide whether a pixel is well described by the background model. The default value is 16.
・detectShadows: A Boolean indicating whether to detect shadows. If set to True, the algorithm detects and marks shadows, which are typically shown in gray in the output mask. The default value is True.

Now using this method, the motion is much brighter with enter bodies being highlighted, we can even see some differences in people who are walking across the crossing. Like someone holding an umbrella. But there is a lot more noise in this version with some background artifacts from frame to frame.

As Shadows are detected in fgbg object, a separate value is set to them than the object which is solid white. we can do this by a simple array boolean change to set all non 255 values to zero

fgmask[fgmask != 255] = 0

There is also still a lot of noize in the version . One way to reduce this is use the value channel from a hsv image then apply a slight blur to cut out those artifacts

def process_frame(frame):
    frame= cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)[:,:,2]


    frame = cv2.GaussianBlur(frame, (5, 5), 1)
    return frame

This reduced the pixel artifact but does not solve them entirely but its good enough for now.

Method 3 Motion Tracking with cv2.calcOpticalFlowPyrLK

In previous sections, we explored motion detection based on changes in pixel values between frames. However, detecting motion alone doesn't provide insights into how individual objects move. OpenCV's cv2.calcOpticalFlowPyrLK is a powerful tool for tracking specific points across video frames. This function uses the Lucas-Kanade method for optical flow estimation, making it highly effective for tracking dynamic objects in scenes such as the bustling traffic at Shibuya crossing.

import cv2
import numpy as np

# Parameters for Shi-Tomasi corner detection
feature_params = dict(maxCorners=100, qualityLevel=0.3, minDistance=7, blockSize=7)

# Parameters for Lucas-Kanade optical flow
lk_params = dict(winSize=(15, 15), maxLevel=2, criteria=(cv2.TERM_CRITERIA_EPS | cv2.TERM_CRITERIA_COUNT, 10, 0.03))

# Video capture and output setup
cap = cv2.VideoCapture("shibuya_traffic.mp4")
fourcc = cv2.VideoWriter_fourcc(*'mp4v')
rec = cv2.VideoWriter("output_tracking.mp4", fourcc, 30.0, (int(cap.get(3)), int(cap.get(4))))

# Read the first frame and initialize tracking points
ret, old_frame = cap.read()
old_gray = cv2.cvtColor(old_frame, cv2.COLOR_BGR2GRAY)
p0 = cv2.goodFeaturesToTrack(old_gray, mask=None, **feature_params)

while True:
    ret, frame = cap.read()
    if not ret:
        break

    # Convert the current frame to grayscale
    frame_gray = cv2.cvtColor(frame.copy(), cv2.COLOR_BGR2GRAY)

    # Calculate optical flow
    p1, st, err = cv2.calcOpticalFlowPyrLK(old_gray, frame_gray, p0, None, **lk_params)

    # Select points successfully tracked
    good_new = p1[st == 1]
    good_old = p0[st == 1]

    # Draw the tracked points
    for i, (new, old) in enumerate(zip(good_new, good_old)):
        a, b = map(int, new.ravel())
        c, d = map(int, old.ravel())
        cv2.circle(frame, (a, b), 5, (0, 0, 255), -1)
        cv2.line(frame, (a, b), (c, d), (0, 255, 0), 2)

    # Update the previous frame and points
    old_gray = frame_gray.copy()
    p0 = good_new.reshape(-1, 1, 2)

    # Write and display the frame
    rec.write(frame)
    cv2.imshow("Tracked Motion", frame)

    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

cap.release()
rec.release()
cv2.destroyAllWindows()

Setting Key Points for Tracking

Before tracking can begin, initial key points must be identified. This is done using the cv2.goodFeaturesToTrack function:

p0 = cv2.goodFeaturesToTrack(old_gray, mask=None, **feature_params)

Parameters for feature_params:

maxCorners: The maximum number of corners (keypoints) to detect. Increasing this value allows more points to be tracked but increases computational cost.
qualityLevel: The minimum accepted quality of detected corners (0 to 1). Lower values detect weaker corners but may reduce tracking reliability.
minDistance: Minimum Euclidean distance between detected points. Smaller values allow detection of closer points, increasing density.
blockSize: Size of the neighborhood considered for corner detection. Smaller values detect finer details but are more sensitive to noise.

Lucas-Kanade Optical Flow Parameters

lk_params = dict(winSize=(15, 15), maxLevel=2, criteria=(cv2.TERM_CRITERIA_EPS | cv2.TERM_CRITERIA_COUNT, 10, 0.03))

winSize: Defines the size of the search window for each pyramid level. Smaller windows increase precision but may miss large-scale motion.
maxLevel: Sets the number of pyramid levels used for optical flow estimation. Higher levels improve tracking over larger displacements but require more computation.
criteria: Determines the stopping condition for the iterative search algorithm:
- EPS: Stops when the search achieves a small enough change.
- COUNT: Stops after a fixed number of iterations.

In a complex scene like Shibuya crossing, adjusting parameters can enhance tracking results. Increasing the number of corners and reducing the search window size can improve motion precision:

# Optimized parameters
feature_params = dict(maxCorners=500, qualityLevel=0.3, minDistance=7, blockSize=7)
lk_params = dict(winSize=(7, 7), maxLevel=5, criteria=(cv2.TERM_CRITERIA_EPS | cv2.TERM_CRITERIA_COUNT, 10, 0.03))

Conclusion

In this article, we explored three powerful motion detection and tracking methods in OpenCV: Frame Delta, Background Subtraction, and Optical Flow using cv2.calcOpticalFlowPyrLK. Each method has its strengths and ideal use cases, from detecting general motion to tracking precise movements of specific objects.

The Frame Delta method provides a straightforward approach for comparing consecutive frames, while Background Subtraction offers a robust way to separate dynamic foreground objects from a static background. Finally, the Optical Flow method enables detailed tracking of motion by following individual points frame-by-frame.

By understanding and implementing these techniques, you can analyze and visualize motion effectively, even in complex scenes like the bustling Shibuya crossing.

Homework

try combining the Frame Delta and Background Subtraction methods with the Optical Flow approach. Here’s your challenge:

Objective:
Enhance motion detection by blending the strengths of the Frame Delta and MOG2 methods to preprocess the frames before applying Optical Flow for tracking.

Message me or leave a comment with your code and output video link. I'd love to see how you combine these methods and the insights you gain from the process!

[CV2] HSV vs RGB: Understanding and Leveraging HSV for Image Processing

Daniel Jarvis — Sat, 30 Nov 2024 03:24:26 +0000

In the previous post, we explored the basics of working with RGB images in OpenCV, including plotting and adjusting brightness and contrast. While the RGB color space is ideal for computer displays, as it represents colors in terms of light intensity emitted by screens, it doesn’t align with how humans perceive colors in the natural world. This is where HSV (Hue, Saturation, Value) steps in—a color space designed to represent colors in a way that's closer to human perception.
In this post, we’ll dive into HSV, understand its components, explore its applications, and learn some cool tricks to enhance images.

What is HSV?

HSV stands for Hue, Saturation, and Value:

Hue (H): This refers to the type of color—red, green, blue, etc. While traditionally measured in degrees on a circular spectrum (0°–360°), in OpenCV, the Hue is scaled to 0–179 to fit within an 8-bit integer. Here's the mapping:

0 (or near it) still represents red.

60–89 corresponds to green.

120–149 corresponds to blue.

140–179 wraps back around to red, completing the circular spectrum.

Saturation (S): This defines the intensity or purity of a color: A fully saturated color contains no gray and is vibrant, A less saturated color appears more washed out.
Value (V): Often referred to as brightness, it measures the lightness or darkness of By separating these components, HSV makes it easier to analyze and manipulate images, especially for tasks like color detection or enhancement. the color.

To understand this better the plot blow is a good presentation of there values in color space

Converting an Image to HSV in OpenCV

Converting an image to HSV in OpenCV is straightforward with the cv2.cvtColor() function. Let’s take a look:

import cv2
import matplotlib.pyplot as plt


image = cv2.imread('./test.png')
plt.figure(figsize=(10,10))
plt.subplot(1,2,1)
plt.imshow(image[:,:,::-1]) #plot as RGB 
plt.title("RGB View")
hsv= cv2.cvtColor(image, cv2.COLOR_RGB2HSV)
plt.subplot(1,2,2)
plt.imshow(hsv)
plt.title("HSV View")
plt.tight_layout()
plt.show()

At first glance, the HSV plot might look strange—almost alien-like. That’s because your computer is trying to represent HSV as an RGB image, even though the components of HSV (especially Hue) aren’t directly mapped to RGB values. For example:

Hue (H): Represented as an angle, it ranges from 0 to 179 in OpenCV (not 0 to 255 like RGB channels). This causes the Hue channel to appear predominantly blue in RGB-based plots.

For the next following examples we not going to use the profile image but a darker image generated with Flux ai image gen model. as it provide a better user case of HSV that the profile image, as we can see its effect better

Understanding HSV Through Histograms

To better understand the differences between RGB and HSV, let’s plot histograms for each channel. Here’s the code:

# Plot the histograms
plt.figure(figsize=(10, 6))

# RGB Histogram
plt.subplot(1, 2, 1)
for i, color in enumerate(['r', 'g', 'b']):
    plt.hist(image[:, :, i].ravel(), 256, [0, 256], color=color, histtype='step')
    plt.xlim([0, 256])
plt.title("RGB Histogram")

# HSV Histogram
plt.subplot(1, 2, 2)
for i, color in enumerate(['r', 'g', 'b']):
    plt.hist(hsv[:, :, i].ravel(), 256, [0, 256], color=color, histtype='step')
    plt.xlim([0, 256])
plt.title("HSV Histogram")
plt.show()

From the histograms, you can see how the HSV channels differ from RGB. Notice the Hue channel in HSV, which has values between 0 and 179, representing distinct color regions, while Saturation and Value handle intensity and brightness.

Visualizing Hue, Saturation, and Value

Now, let’s break the HSV image into its individual components to better understand what each channel represents:

# Plot the individual HSV channels
plt.figure(figsize=(10, 6))
plt.subplot(1, 3, 1)
plt.imshow(hsv[:, :, 0], cmap='hsv')  # Hue
plt.title("Hue")
plt.subplot(1, 3, 2)
plt.imshow(hsv[:, :, 1], cmap='gray')  # Saturation
plt.title("Saturation")
plt.subplot(1, 3, 3)
plt.imshow(hsv[:, :, 2], cmap='gray')  # Value
plt.title("Value")
plt.tight_layout()
plt.show()

Hue: Displays clear color distinctions, highlighting the dominant colors in the image.
Saturation: Brighter areas represent vibrant colors, while darker areas indicate more muted, grayish tones.
Value: Highlights the brightness distribution, with well-lit areas appearing brighter.

Tricks with HSV

1. Brightness Enhancement (Value Equalization)

For images with uneven lighting, equalizing the Value channel can make darker areas more visible while giving a "glow" effect to brighter regions.

equ = cv2.equalizeHist(hsv[:, :, 2])  # Equalize the Value channel
new_hsv = cv2.merge((hsv[:, :, 0], hsv[:, :, 1], equ))
new_image = cv2.cvtColor(new_hsv, cv2.COLOR_HSV2BGR)

# Display results
plt.figure(figsize=(10, 6))
plt.subplot(1, 2, 1)
plt.imshow(image)
plt.title("Original Image")
plt.subplot(1, 2, 2)
plt.imshow(new_image)
plt.title("Brightness Enhanced")
plt.tight_layout()
plt.show()

2. Color Enhancement (Saturation Equalization)

Boosting the Saturation channel makes colors in the image more distinct and vibrant.

equ = cv2.equalizeHist(hsv[:, :, 1])  # Equalize the Saturation channel
new_hsv = cv2.merge((hsv[:, :, 0], equ, hsv[:, :, 2]))
new_image = cv2.cvtColor(new_hsv, cv2.COLOR_HSV2BGR)

# Display results
plt.figure(figsize=(10, 6))
plt.subplot(1, 2, 1)
plt.imshow(image)
plt.title("Original Image")
plt.subplot(1, 2, 2)
plt.imshow(new_image)
plt.title("Color Enhanced")
plt.tight_layout()
plt.show()

3. Color Filtering (Isolating Red)

Using the Hue channel, we can isolate specific colors. For example, to extract red tones:

# Define range for red color
lower_red = np.array([140, 0, 0])
upper_red = np.array([180, 255, 255])

# Create a mask
mask = cv2.inRange(hsv, lower_red, upper_red)
filtered_image = cv2.bitwise_and(image, image, mask=mask)

# Display results
plt.figure(figsize=(10, 6))
plt.subplot(1, 2, 1)
plt.imshow(image)
plt.title("Original Image")
plt.subplot(1, 2, 2)
plt.imshow(filtered_image)
plt.title("Red Filtered")
plt.tight_layout()
plt.show()

This technique is incredibly useful for tasks like object detection, color segmentation, or even artistic effects.

Conclusion

The HSV color space offers a versatile and intuitive way to analyze and manipulate images. By separating color (Hue), intensity (Saturation), and brightness (Value), HSV simplifies tasks like color filtering, enhancement, and segmentation. While RGB is ideal for displays, HSV opens up possibilities for creative and analytical image processing.

What’s your favorite trick with HSV? Share your thoughts below, and let’s explore this vibrant world of color together!

This version incorporates a smooth flow, detailed explanations, and consistent formatting to improve readability and comprehension.

CV2: What is an Image?, Lets adjust the Brightness and Contrast of an image

Daniel Jarvis — Fri, 29 Nov 2024 02:11:58 +0000

In the context of CV2 in Python, when you read an image, it is stored as a 2D or 3D array of Y and X coordinates with uint8 values indicating the color and brightness of the image. The term uint8 refers to an 8-bit unsigned integer data type that ranges from 0 to 255. This, combined with 3 channels for Red, Green, and Blue (RGB), forms a color image.

Why This Structure Matters

If you start altering parts of the image like a normal array (e.g., dividing it by 3), you may lose this format. For instance, pixel values may fall outside the range of 0 to 255, causing the image to become unusable. Understanding this structure is crucial for manipulating images correctly.

Viewing an Image in CV2

In the CV2 Python library, you can easily view an image using the following code snippet:

#pip install opencv-python # if not already installed 
import cv2

# Load an image
image = cv2.imread('./test.png')

# Display the image in a window
cv2.imshow('Image', image)
cv2.waitKey(0)
cv2.destroyAllWindows()

The code above will open a popup window displaying the image on your computer. Feel free to add a url to your own image to test. You can then zoom in to observe the pixel-level RGB values of the image. This basic functionality is a great starting point for exploring image processing.

Brightness and Contrast

What Are Brightness and Contrast?

Brightness refers to the overall lightness or darkness of an image, determined by the pixel intensity.
Contrast refers to the difference in intensity between pixels compared to a reference value (e.g., the mean pixel intensity). It essentially measures how "sharp" or "distinct" the variations in the image are.

Mathematically, brightness and contrast can be adjusted using the formula:

new_image=contrast×image+brightness

Applying Brightness and Contrast in CV2

The cv2.convertScaleAbs() function in OpenCV automates this process. It applies the formula above while ensuring that pixel values remain within the range of 0 to 255.

Here’s how it works:

alpha (contrast): A scaling factor, typically between 0.0 and 3.0.
beta (brightness): An offset value, typically between -100 and 100

Example Usage:
new_image = cv2.convertScaleAbs(image, alpha=contrast, beta=brightness)
This allows us to easily modify brightness and contrast without manually clipping pixel values.

image = cv2.imread('./test.png')
cv2.namedWindow('Adjustments')
contrast=0.8
brightness=89
image=cv2.convertScaleAbs(image, alpha=contrast, beta=brightness)
cv2.imshow('Adjustments', image) 
cv2.waitKey(0)
cv2.destroyAllWindows()

Creating Interactive Adjustments with Callbacks

While a one-time adjustment is useful, most of the time, we want to tweak brightness and contrast interactively. OpenCV allows us to achieve this using trackbar.

Trackbars can be created with cv2.createTrackbar(), which lets us adjust values dynamically. The general syntax is:
cv2.createTrackbar(trackbarname, winname, value, count, onChange_function)

trackbarname: The name of the trackbar.
winname: The name of the OpenCV window where the trackbar will be displayed.
value: The initial position of the trackbar.
count: The maximum value of the trackbar.
onChange_function: A callback function that is called whenever the trackbar value changes.

These trackbar then can be called in the onChange_function with;
cv2.getTrackbarPos(trackbarname, winname)
・trackbarname: The name of the trackbar
・ winname: The name of the OpenCV window where the trackbar will be displayed.

To adjust both brightness and contrast, we will need two trackbars.


def on_change(*args):
    alpha = cv2.getTrackbarPos('Contrast', 'Adjustments') / 10 #[0.0, 3.0]
    beta = cv2.getTrackbarPos('Brightness', 'Adjustments') - 50  #[-100, 100].
    updated_image = cv2.convertScaleAbs(image, alpha=alpha, beta=beta)
    cv2.imshow('Adjustments', updated_image)

# Display the image
image = cv2.imread('./test.png')
cv2.namedWindow('Adjustments')
cv2.imshow('Adjustments', image) 

# Add trackbars
cv2.createTrackbar('Contrast', 'Adjustments', 10, 30, on_change)  # Contrast 1.0 to 3.0
cv2.createTrackbar('Brightness', 'Adjustments', 50, 100, on_change)  # Brightness -50 to 50


cv2.imshow('Adjustments', image) 
cv2.waitKey(0)
# Exit on pressing 'x'
while True:
    key = cv2.waitKey(1) & 0xFF
    if key == ord('x'):  # Press 'x' to exit
        print("Exiting...")
        break
cv2.destroyAllWindows()

Adjusting the contrast and brightness sliders triggers the on_change function, which reads the values from the trackbars using cv2.getTrackbarPos(). These values are then applied to the image using the cv2.convertScaleAbs function, and the updated image is displayed in real-time.

To make the app more user-friendly, I added a simple snippet at the end to allow users to exit by pressing the x key. This addresses a common issue with OpenCV, where closing the window doesn’t always stop the code from running. By implementing this, the app ensures a clean exit without lingering processes.