DEV Community: Dhaarani Dhanasekeran

A Beginner Project with Computer Vision and Python

Dhaarani Dhanasekeran — Wed, 04 Nov 2026 15:23:17 +0000

Introduction

Computer vision is a fascinating field that enables software to interpret and understand images and videos. It has applications ranging from security cameras to self-driving cars. In this article, we'll explore a simple project using computer vision and Python, starting with the basics and moving on to more advanced concepts.

Understanding Computer Vision

Computer vision involves several key concepts such as image processing, feature extraction, and object recognition. These techniques allow software to analyze images and videos and extract meaningful information from them.

Using OpenCV for Image Processing

OpenCV is a powerful library for computer vision tasks. It provides various modules for image and video processing. For our project, we'll use OpenCV to read and display images.

A Simple Image Processing Project

Let's create a simple project that reads an image, applies a grayscale filter, and displays the result.

import cv2

def process_image(image_path):
    # Read the image using OpenCV
    image = cv2.imread(image_path)

    # Convert the image to grayscale
    gray_image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)

    # Display the grayscale image
    cv2.imshow('Grayscale Image', gray_image)
    cv2.waitKey(0)
    cv2.destroyAllWindows()

# Example usage
process_image('path/to/image.jpg')

In this code, we first import the cv2 module from OpenCV. We define a function process_image that reads an image from a specified path, converts it to grayscale, and then displays the grayscale image using OpenCV's imshow function. The waitKey function waits for a key press before closing the image window, and destroyAllWindows closes all OpenCV windows.

Feature Extraction

Feature extraction is the process of identifying distinctive features in images. For example, we might want to detect edges, corners, or specific patterns in an image. OpenCV provides several feature extraction methods, such as SIFT (Scale-Invariant Feature Transform) and SURF (Speeded-Up Robust Features).

Object Detection with Haar Cascades

Haar cascades are a popular method for object detection. They work by identifying characteristic features in images, such as edges and corners, and using these features to detect objects. OpenCV provides a pre-trained cascade classifier for face detection.

import cv2

def detect_faces(image_path):
    # Read the image
    image = cv2.imread(image_path)

    # Load the pre-trained cascade classifier for face detection
    face_cascade = cv2.CascadeClassifier(cv2.data.haarcascades + 'haarcascade_frontalface_default.xml')

    # Detect faces in the image
    faces = face_cascade.detectMultiScale(image, scaleFactor=1.1, minNeighbors=5, minSize=(30, 30), flags=cv2.CASCADE_SCALE_IMAGE)

    # Draw rectangles around the detected faces
    for (x, y, w, h) in faces:
        cv2.rectangle(image, (x, y), (x+w, y+h), (255, 0, 0), 2)

    # Display the image
    cv2.imshow('Detected Faces', image)
    cv2.waitKey(0)
    cv2.destroyAllWindows()

# Example usage
detect_faces('path/to/image.jpg')

In this code, we load the pre-trained cascade classifier for face detection, which is stored in a pre-defined path. We then use the detectMultiScale method to detect faces in the image. The method returns a list of bounding boxes for detected faces. We draw rectangles around these boxes for visualization.

Handling Common Pitfalls

Working with OpenCV and computer vision tasks can sometimes be challenging. Here are some common pitfalls to watch out for:

Image Format Issues: Ensure that the image format matches the one expected by OpenCV, such as BGR for color images. If the image is in a different format, you may need to convert it.
Feature Extraction Precision: SIFT and SURF can sometimes be sensitive to variations in lighting and pose. Experiment with different feature extraction methods and adjust parameters as needed.
Object Detection Accuracy: Object detection methods like Haar cascades may struggle with objects that do not match the pre-trained templates. Consider using more advanced techniques or training your own models if necessary.

Conclusion

In this article, we explored the basics of computer vision using OpenCV and Python. We saw how to read and process images, as well as how to detect objects using Haar cascades. These are fundamental skills for any project involving computer vision. With practice and experimentation, you can expand your knowledge and build more complex applications.

Additional Resources

Data Leakage in Machine Learning: Examples and Fixes

Dhaarani Dhanasekeran — Wed, 04 Nov 2026 15:22:42 +0000

Introduction

Data leakage is a common issue in machine learning where sensitive or training-specific data is unknowingly included in the testing or validation datasets. This can lead to overly optimistic performance metrics and poor generalization to new, unseen data. In this article, we'll explore how data leakage occurs, provide concrete examples, and offer solutions to prevent it.

Understanding Data Leakage

Data leakage can manifest in several ways, such as:

Training Data Leaks into Validation or Test Sets: During model training, data from the training set unintentionally leaks into the validation or test set.
Use of Sensitive Data: Using data that should not be part of the training set, such as personally identifiable information (PII), timestamps, or specific IDs.
Overfitting to Particular Data: The model performs well on the part of the data it was trained on but fails on data it has not seen during training.

Common Pitfalls and Examples

Example: Training Data Leaks into Test Set

Consider a scenario where we're developing a spam classifier. We use the spam dataset from the sklearn library, which contains emails labeled as spam or not spam. During testing, we realize our model is performing poorly on some emails marked as spam. Upon investigation, we find that some emails in the test set also have timestamps and sender IDs. These details should not be part of the training data and are causing the model to overfit to specific characteristics of the emails.

Here is the code snippet that demonstrates this issue:

from sklearn.datasets import fetch_spam_emails
from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.metrics import accuracy_score

# Fetch spam emails dataset
spam_emails = fetch_spam_emails()
emails = spam_emails.data
labels = spam_emails.target

# Train-test split
X_train, X_test, y_train, y_test = train_test_split(emails, labels, test_size=0.2, random_state=42)

# Vectorize emails
vectorizer = CountVectorizer()
X_train_vec = vectorizer.fit_transform(X_train)
X_test_vec = vectorizer.transform(X_test)

# Train a model
model = MultinomialNB()
model.fit(X_train_vec, y_train)

# Predict and evaluate
y_pred = model.predict(X_test_vec)
print('Accuracy:', accuracy_score(y_test, y_pred))

In the above code, the CountVectorizer does not include any features like timestamps or sender IDs. However, if these features were present in the test set, they could cause the model to overfit. This is a simple example, but in larger datasets, such issues can be more subtle and harder to detect.

Pitfalls to Avoid

Careful Data Preprocessing: Ensure all features are correctly preprocessed. For example, if timestamps are included in the dataset, ensure they are normalized or transformed properly.
Data Masking: Use techniques like masking or removing certain columns or rows containing sensitive data.
Feature Engineering: Avoid creating features that are highly correlated with the target variable if they are not meaningful in the real-world scenario.

Concrete Example Solution

Let's revisit the spam classifier example with a solution to mitigate data leakage. We'll use the imbalanced-learn library to remove sensitive features like timestamps and sender IDs.

from imblearn.under_sampling import RandomUnderSampler
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import MultinomialNB
from sklearn.metrics import accuracy_score
from imblearn.pipeline import Pipeline

# Fetch spam emails dataset
spam_emails = fetch_spam_emails()
emails = spam_emails.data
labels = spam_emails.target

# Train-test split
X_train, X_test, y_train, y_test = train_test_split(emails, labels, test_size=0.2, random_state=42)

# Data masking
# Remove sensitive features like timestamps and sender IDs
# This is a placeholder for actual masking logic
X_train = [email.replace('@example.com', '') for email in X_train]
X_test = [email.replace('@example.com', '') for email in X_test]

# Vectorize emails
vectorizer = TfidfVectorizer()
X_train_vec = vectorizer.fit_transform(X_train)
X_test_vec = vectorizer.transform(X_test)

# Train a model
model = MultinomialNB()
model.fit(X_train_vec, y_train)

# Predict and evaluate
y_pred = model.predict(X_test_vec)
print('Accuracy:', accuracy_score(y_test, y_pred))

Here, we've replaced email addresses with a placeholder to remove the sensitive information. This is a simplified example, and in real-world scenarios, more sophisticated masking techniques may be needed to handle various types of sensitive data.

Conclusion

Data leakage is a significant challenge in machine learning that can lead to suboptimal model performance. By understanding the common sources of data leakage and employing robust strategies to prevent it, developers can ensure their models generalize well to unseen data. By maintaining rigorous data preprocessing and using techniques like masking, we can mitigate the risks associated with data leakage and build more reliable machine learning models.

Tags

machine-learning
data-leakage
model-optimization

Description

This article discusses the issue of data leakage in machine learning, its impact, and provides examples and solutions to prevent it, ensuring models generalize well to unseen data.

Introduction to Graph Databases for Knowledge Graphs

Dhaarani Dhanasekeran — Wed, 04 Nov 2026 15:17:43 +0000

Introduction

Graph databases are specialized database management systems that store and manage data as interconnected nodes and relationships. They are particularly well-suited for creating and querying knowledge graphs, which are complex data structures used to represent and analyze relationships within domains such as social networks, knowledge management systems, and semantic web technologies. In this article, we'll explore the basics of graph databases and their application to knowledge graphs.

What is a Graph Database

At their core, graph databases store data as nodes (vertices) and relationships (edges) between those nodes. Each node can have properties associated with it, and each relationship can also have properties. This structure allows for efficient querying of complex data relationships and enables the representation of large and diverse datasets in a unified way. Common examples of graph databases include Neo4j, Amazon Neptune, and JanusGraph.

Knowledge Graphs

Knowledge graphs are an advanced form of graph databases that are designed to represent and analyze complex real-world domains. They aim to capture the intricate relationships and knowledge within these domains, making them valuable tools for data integration, recommendation systems, and semantic search.

Example: Building a Knowledge Graph

Let's illustrate how graph databases can be used to build a simple knowledge graph. In this example, we'll create a knowledge graph to represent a movie recommendation system. We'll use Neo4j, a popular graph database, to store and query data.

Creating the Movie Nodes

First, we'll create nodes representing movies. Each movie node will have properties such as title, director, year, and genre. Here's how you can create these nodes in Neo4j's Cypher query language:

CREATE (movie1:Movie {title: 'Inception', director: 'Christopher Nolan', year: 2010, genre: 'Sci-Fi'})
CREATE (movie2:Movie {title: 'The Dark Knight', director: 'Christopher Nolan', year: 2008, genre: 'Action'})
CREATE (movie3:Movie {title: 'Interstellar', director: 'Christopher Nolan', year: 2014, genre: 'Sci-Fi'})

Creating the Relationship Nodes

Next, we'll create relationships between the movie nodes to represent connections such as being directed by or starring with another movie. Here's an example of how to create a relationship from movie1 to movie2 (movie2 directed movie1):

MATCH (m1:Movie {title: 'Inception'}), (m2:Movie {title: 'The Dark Knight'}), (m3:Movie {title: 'Interstellar'})
CREATE (m1)-[:DIRECTED_BY]->(m2)

Querying the Knowledge Graph

Now that we have our movie nodes and relationships, we can query the knowledge graph to find related movies. For instance, we can find all movies directed by Christopher Nolan:

MATCH (m1:Movie {director: 'Christopher Nolan'})
RETURN m1

This query will return all movies in our graph where the director is 'Christopher Nolan'.

Common Pitfalls

While graph databases are powerful, they come with their own set of challenges and potential issues, such as data consistency and integrity, query optimization, and schema design. Ensuring that your graph database is well-structured and optimized for your use case is crucial for maintaining performance and data quality.

Conclusion

Graph databases provide a robust framework for building and querying complex knowledge graphs. By leveraging their unique strengths in handling interrelated data, graph databases can significantly enhance the capabilities of applications requiring advanced data analysis and querying. Whether you're working on recommendation systems, semantic search, or any application that benefits from a deep understanding of relationships within data, graph databases are a valuable tool to consider.

tags: graph-databases, knowledge-graphs, neo4j, cypher, data-relationships, semantic-search, data-integration
description: An introduction to graph databases for building knowledge graphs, with a focus on Neo4j and Cypher for querying and managing data relationships.

Making AI-Generated Code Easier to Review

Dhaarani Dhanasekeran — Wed, 04 Nov 2026 15:17:09 +0000

Introduction

AI-generated code has the potential to streamline development workflows, but it can also introduce complexity when reviewing the generated code. In this article, we'll explore strategies to make AI-generated code more accessible and understandable for human reviewers. We'll cover practical techniques to improve the quality and readability of AI-generated code.

Understanding AI-Generated Code

AI-generated code is often generated from natural language descriptions or through code generation models. This can result in code that lacks documentation, has complex structures, and may not follow conventional programming standards. As a developer, it's crucial to understand these characteristics to effectively review AI-generated code.

Structuring AI-Generated Code

One approach to making AI-generated code more reviewable is by structuring the code in a way that mirrors conventional programming practices. This includes using clear naming conventions, organizing code into logical blocks, and including comments and documentation where appropriate. For example, consider the following AI-generated code snippet:

# AI-generated code
async def generate_report(data):
    response = await client.get_report(data)
    return response

This code is straightforward but lacks documentation and context. By refactoring the code with better structure, it becomes more understandable. Here's an improved version:

# Structured AI-generated code
async def generate_report(data):
    # Fetch the report from the client
    response = await client.get_report(data)

    # Return the fetched response
    return response

Structured code with clear comments and names improves readability and maintainability.

Including Comments and Documentation

Adding comments and documentation to AI-generated code is essential for reviewers. Comments should explain why certain decisions were made, and documentation should provide context about the code's purpose and functionality. For instance, if an AI model generates a complex SQL query, including comments explaining the logic behind the query can be immensely helpful:

# AI-generated SQL query
sql_query = f"SELECT * FROM {table} WHERE {condition};"

# Structured code with comments
sql_query = f"SELECT * FROM {table} WHERE {condition};"  # Selects all rows from the table where condition is met

# Documentation
"""
Query to select all rows from the table where the condition is met.

Parameters:
- table (str): The name of the table to query.
- condition (str): The condition to filter the rows.

Returns:
- list: A list of rows from the table that match the condition.
"""

By including comments and documentation, reviewers can better understand the code's intent and functionality.

Identifying and Fixing Common Pitfalls

AI-generated code can introduce several common pitfalls, such as over-complicated logic, unnecessary complexity, and lack of maintainability. Identifying and addressing these issues is crucial for ensuring the quality of AI-generated code. For example, AI models might generate overly complex loops or unnecessary conditional statements. By reviewing and refining these sections, the code can be made more readable and maintainable. Consider the following example where an AI model generated a loop that iterates over an array multiple times unnecessarily:

# AI-generated loop
for item in items:
    for nested_item in item:
        # Perform some operation

To improve readability, refactor the code to eliminate the unnecessary nested loop:

# Refactored loop
for item in items:
    # Perform some operation

In this case, the nested loop is no longer necessary, and the code becomes cleaner and more maintainable.

Conclusion

Making AI-generated code more reviewable is a continuous process that requires a combination of structured code, clear comments, and effective documentation. By applying these strategies, developers can enhance the quality and understandability of AI-generated code, making it easier to review and maintain. As AI technologies continue to evolve, understanding and improving the reviewability of AI-generated code will be increasingly important for developers.

Retries, Timeouts, and Idempotency in API Clients

Dhaarani Dhanasekeran — Wed, 04 Nov 2026 14:45:10 +0000

Introduction

When building software that interacts with external APIs, it's crucial to design resilient clients that can handle failures gracefully. This includes implementing retries, timeouts, and ensuring operations are idempotent. In this article, we'll discuss these concepts in detail, along with practical examples and common pitfalls to avoid.

Retries

Retries are a fundamental strategy for handling transient failures, such as network errors or rate limits. However, they need to be implemented carefully to avoid overwhelming the server or causing data inconsistencies.

Timeouts

Timers are essential for preventing long-running requests from holding up the client. Setting reasonable timeouts helps in managing the client's resources and avoiding potential issues like deadlocks or memory leaks.

Idempotency

Idempotent operations are those that can be performed multiple times without changing the outcome. Ensuring that API calls are idempotent allows for retries and retries with different parameters (e.g., retrying with updated data) without unintended consequences.

Concrete Example

Let's consider a simple API client that fetches user data. Below is an example of how to implement retries, timeouts, and idempotency.

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
from requests.exceptions import RequestException
from time import sleep

# Define our API endpoint
API_ENDPOINT = 'https://example.com/api/user'

# Define our retry strategy
retries = Retry(total=3, backoff_factor=0.1, status_forcelist=[500, 502, 503, 504])
adapter = HTTPAdapter(max_retries=retries)
session = requests.Session()
session.mount('http://', adapter)
session.mount('https://', adapter)

# Function to fetch user data
def fetch_user_data(user_id, retry_count=0):
    # Define our idempotent request
    request = session.get(f'{API_ENDPOINT}/{user_id}', timeout=60)
    if request.status_code == 200:
        print('User data fetched successfully.')
        return request.json()
    elif request.status_code == 429:
        # Rate limit exceeded, retry with backoff
        if retry_count < 3:
            print('Rate limit exceeded, retrying...')
            sleep(1)  # Wait for a second
            return fetch_user_data(user_id, retry_count + 1)
        else:
            print('Exceeded maximum retries, giving up.')
            raise RequestException('API call failed due to rate limit and max retries.')
    else:
        # Handle other failures
        print(f'Unexpected status code: {request.status_code}, retrying...')
        return fetch_user_data(user_id, retry_count)

# Fetch user data for a specific user
try:
    user_data = fetch_user_data('1234567890')
    print(f'User data: {user_data}')
except RequestException as e:
    print(f'Error fetching user data: {e}')

Common Pitfalls

Ignoring Retries: Not implementing retries can lead to unhandled transient failures, causing your application to crash or become unstable.
Setting Inadequate Timeouts: An excessively short timeout can cause your application to timeout frequently, while a timeout that is too long can lead to deadlocks or resource leaks.
Not Considering Idempotency: Calling an idempotent API with different parameters (e.g., retrying with updated data) can lead to unintended data modifications.

Conclusion

By carefully implementing retries, timeouts, and idempotency in your API clients, you can build more robust and reliable software. These practices help you handle errors gracefully and ensure the integrity of your application's state.

Tags

api-clients
retries
timeouts
idempotency

Description

An in-depth look at the importance of retries, timeouts, and idempotency in API clients, with practical examples and common pitfalls to avoid.

Structuring a Maintainable React and FastAPI Project

Dhaarani Dhanasekeran — Wed, 04 Nov 2026 14:44:37 +0000

Introduction

When building a full-stack application that uses both React for the frontend and FastAPI for the backend, it's crucial to structure the project in a way that ensures maintainability and scalability. This guide will walk you through structuring your React and FastAPI project effectively.

Section 1: Frontend-Backend Communication

In a React and FastAPI project, the frontend will interact with the backend through API calls. It’s important to define an API schema that both the frontend and backend can adhere to. Start by defining the API schema in a separate module that both the React frontend and FastAPI backend can import.

Section 2: Separation of Concerns

Apply the principle of separation of concerns to ensure each component or service in your project is responsible for a single job. For example, the React frontend should handle user interfaces and event handling, while the FastAPI backend should handle business logic and API endpoints.

Section 3: Modular Code Structure

Organize your code into smaller, more manageable modules. For instance, you could have a services directory for service layer components, a components directory for reusable UI components, and a utils directory for utility functions that might be used across multiple files. This structure makes your codebase easier to navigate and modify.

Section 4: Using TypeScript for Type Safety

Consider using TypeScript for your project. It can significantly enhance code quality and readability by providing static type checking. By defining types for your API responses and errors, you ensure that your frontend and backend are aligned.

Section 5: Example: Implementing a User Authentication API

Here’s an example of how you might set up a user authentication API using FastAPI and a User model:

from fastapi import FastAPI, Depends, HTTPException
from pydantic import BaseModel

User = dict[str, str]

class UserAuthRequest(BaseModel):
    username: str
    password: str

app = FastAPI()

@app.post("/auth", response_model=User)
async def authenticate_user(auth_request: UserAuthRequest, user_repo: UserRepository = Depends(user_repo_dependency)) -> User:
    user = user_repo.authenticate(auth_request.username, auth_request.password)
    if not user:
        raise HTTPException(status_code=401, detail="Invalid credentials")
    return user

This API call expects an auth_request object with username and password fields. The User type ensures that the user_repo.authenticate method receives the correct type of object.

Section 6: Avoiding Common Pitfalls

Overcomplicating the API: Keep your API simple and focused. Avoid overloading the API with too many endpoints or complex paths.
Not Enough Documentation: Document your API thoroughly. Use Swagger or other tools to generate API documentation.
Lack of Error Handling: Implement robust error handling in your APIs to provide meaningful error messages and to prevent unexpected crashes.

Conclusion

By following the guidelines provided in this article, you can structure your React and FastAPI project in a way that is maintainable and scalable. This structure not only helps you to maintain your project as it grows but also makes it easier to extend and update. As your application evolves, the structured approach will allow you to add new features and maintain old ones without encountering significant technical debt.

Model Quantization for Local AI: A Practical Guide

Dhaarani Dhanasekeran — Wed, 04 Nov 2026 14:39:43 +0000

Introduction

Model quantization is a technique used to reduce the precision of a machine learning model, often from 32-bit floating point (FP32) to 8-bit integer (INT8), to save memory and speed up inference. This guide will explain the concept of quantization, its benefits, and how to implement it in a local AI environment. We'll walk through a concrete example and address common pitfalls along the way.

Benefits of Quantization

Memory Savings: Reduced model size, enabling deployment on resource-constrained devices.
Speed Improvement: Accelerated inference as INT8 operations are generally faster than FP32.
Energy Efficiency: Lower power consumption for devices that rely on inference.

Practical Implementation

Let's explore how to quantize a simple linear regression model using Python and TensorFlow.

Step 1: Prepare the Model

We'll start with a basic linear regression model. Here is a minimal example of a TensorFlow model:

import tensorflow as tf

model = tf.keras.models.Sequential([
    tf.keras.layers.Dense(1, input_shape=(1,), activation='linear')
])

model.compile(optimizer='adam', loss='mean_squared_error')

Step 2: Quantize the Model

Quantization involves converting the weights and activations of the model to lower precision. TensorFlow provides a quantize API to facilitate this process. Here's how you can quantize the model:

import tensorflow_model_optimization as tfmot

quantize_model = tfmot.quantization.keras.quantize_model

quantize_model = quantize_model(model)

# Prepare the model for training with quantization
quantize_model.compile(optimizer='adam', loss='mean_squared_error')

# Load your dataset
(x_train, y_train), (x_test, y_test) = tf.keras.datasets.boston_housing.load_data()

# Quantize the model
quantize_model.quantize()

# Train the quantized model
quantize_model.fit(x_train, y_train, epochs=100)

# Save and export the quantized model
quantize_model.save('quantized_model.h5')

Step 3: Evaluate the Model

After quantization, it's important to evaluate the model's performance and compare it with the original model. Here's how you might do that:

from sklearn.metrics import mean_squared_error

# Load the quantized model
quantized_model = tf.keras.models.load_model('quantized_model.h5', custom_objects={'KerasLayer': tf.keras.layers.InputLayer})

# Evaluate the quantized model
y_pred_quantized = quantized_model.predict(x_test)

mse_quantized = mean_squared_error(y_test, y_pred_quantized)

# Compare with the original model
original_model = tf.keras.models.load_model('original_model.h5')

y_pred_original = original_model.predict(x_test)

mse_original = mean_squared_error(y_test, y_pred_original)

print(f'Mean Squared Error (Quantized): {mse_quantized:.2f}, Original: {mse_original:.2f}')

Common Pitfalls

Model Loss: Quantization can lead to a small decrease in performance. It's crucial to monitor this and ensure the model's accuracy is not too compromised.
Precision Requirements: The quantization process may require careful tuning of parameters like quantization levels and weights to achieve optimal results.
Compatibility: Some libraries or environments might not support quantized models. Ensure your runtime environment is compatible with quantization.

Conclusion

Model quantization is a powerful technique for optimizing machine learning models for local AI deployment. By reducing precision, you can achieve significant memory and performance benefits. This guide provided a step-by-step walkthrough of quantizing a TensorFlow model, along with practical tips to avoid common pitfalls. As you apply these techniques, consider the specific needs of your application to maximize the benefits of quantization.

Tags

model-quantization
machine-learning
tensorflow
local-ai

Description

A comprehensive guide to model quantization for optimizing machine learning models for local AI environments, including practical implementation steps, benefits, and pitfalls.

Deploying a Small Python API Responsibly

Dhaarani Dhanasekeran — Wed, 04 Nov 2026 14:39:08 +0000

Introduction

Deploying a Python API can be a straightforward process, but it's crucial to do so responsibly to ensure security, scalability, and maintainability. This article covers best practices for deploying a small Python API, focusing on responsible deployment. We'll discuss the importance of authentication, environment management, and logging.

1. Setting Up Authentication

Authentication is the foundation of a secure API. Use a library like flask-security to handle authentication. This library provides user authentication and roles management. Here's a quick example of setting up authentication in a Flask app using flask-security:

from flask import Flask, jsonify
from flask_security import Security, SQLAlchemyUserDatastore, UserMixin, RoleMixin, login_required

app = Flask(__name__)
app.config['SECRET_KEY'] = 'your_secret_key'
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///your_db.db'

db = SQLAlchemy(app)

roles_users = db.Table(
    'roles_users',
    db.Column('user_id', db.Integer(), db.ForeignKey('user.id')),
    db.Column('role_id', db.Integer(), db.ForeignKey('role.id'))
)

class Role(db.Model, RoleMixin):
    __tablename__ = 'role'

    id = db.Column(db.Integer(), primary_key=True)
    name = db.Column(db.String(80), unique=True)
    description = db.Column(db.String(255))

class User(db.Model, UserMixin):
    __tablename__ = 'user'
    id = db.Column(db.Integer, primary_key=True)
    email = db.Column(db.String(255), unique=True)
    password = db.Column(db.String(255))
    active = db.Column(db.Boolean())
    roles = db.relationship('Role', secondary=roles_users, backref=db.backref('users', lazy='dynamic'))

user_datastore = SQLAlchemyUserDatastore(db, User, Role)
security = Security(app, user_datastore)

@app.route('/register', methods=['POST'])
@security.login_required
def register():
    new_user = user_datastore.create_user(email='test@example.com', password='password')
    user_datastore.commit()
    return jsonify({'message': 'User created'}), 201

@app.route('/login', methods=['POST'])
@security.login_required
@login_required
@user_datastore.has_role('admin')
def login():
    user_datastore.commit()
    return jsonify({'message': 'Logged in'}), 200

if __name__ == '__main__':
    app.run()

This example sets up a simple Flask app with user authentication. Users can register and login, and admins have access to a specific route.

2. Environment Management

Managing different environments like development, testing, and production is crucial. Use environment variables for sensitive information such as database URLs and API keys. Libraries like python-dotenv can help with this. Here's how to use python-dotenv to manage environment variables:

from dotenv import load_dotenv
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
from flask_security import SQLAlchemyUserDatastore

app = Flask(__name__)
load_dotenv()

app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///your_db.db' if not app.config['DEVELOPMENT'] else 'sqlite:///dev.db'
app.config['SECRET_KEY'] = 'your_secret_key' if not app.config['DEVELOPMENT'] else 'development_secret_key'

app.config.from_object(__name__)

user_datastore = SQLAlchemyUserDatastore(SQLAlchemy(app), User, Role)
security = Security(app, user_datastore)

if not app.config['DEVELOPMENT']:
    user_datastore.init_app(app)
    db = SQLAlchemy(app)

if __name__ == '__main__':
    app.run()

In this example, the application checks the DEVELOPMENT environment variable to determine the database URI. This prevents sensitive information from leaking to production.

3. Logging and Monitoring

Logging helps in debugging issues and monitoring application health. Use a logging library like flask-log-request-id to include request IDs in logs. This library automatically adds a request ID to logs, making it easier to trace issues across requests. Here's how to use flask-log-request-id:

from flask import Flask, request
from flask_log_request_id import FlaskLogRequestId

app = Flask(__name__)
app.config['DEVELOPMENT'] = True

log_request_id = FlaskLogRequestId(app)

@app.route('/api', methods=['GET'])
def api():
    app.logger.info('Request ID: %s', request.headers.get('X-Request-Id', request.id))
    # Your API logic
    return jsonify({'message': 'API call successful'}), 200

This example logs the request ID from the client header in the API endpoint.

4. Handling Common Pitfalls

Security: Avoid using hard-coded credentials. Instead, use environment variables and secrets management services like HashiCorp Vault or AWS Secrets Manager.
Rate Limiting: Implement rate limiting to prevent abuse and ensure fair usage of your API. Libraries like flask-limiter can help.
Error Handling: Provide meaningful error messages and handle exceptions gracefully. This includes logging errors and returning appropriate HTTP status codes.
Rate Limiting: Implement rate limiting to prevent abuse and ensure fair usage of your API. Libraries like flask-limiter can help.

5. Conclusion

Deploying a Python API requires careful consideration of security, environment management, and logging. By setting up authentication, managing environments, and implementing logging and error handling, you can build a robust and secure API. Remember to follow best practices to ensure your API is reliable and scalable.

6. Further Reading

Flask-Security for user authentication and role management.
python-dotenv for managing environment variables.
flask-limiter for rate limiting.

How to Test FastAPI Endpoints

Dhaarani Dhanasekeran — Wed, 04 Nov 2026 14:33:54 +0000

Introduction

FastAPI is a modern Python web framework for building APIs. Testing FastAPI endpoints is crucial for ensuring that your application behaves as expected. This article will cover the basics of testing FastAPI endpoints, including how to set up tests, common testing techniques, and common pitfalls to avoid.

Setting Up Tests

Before we dive into testing, make sure your FastAPI application has a testing setup. Typically, you'll need to run your FastAPI application in a development server (e.g., uvicorn) and use a testing framework like pytest or unittest.

Testing Basics

Request and Response Assertions

The most basic tests involve making requests to your API and asserting that the response matches your expectations. Here's an example using pytest and uvicorn:

# test_app.py
from fastapi import FastAPI
import pytest

app = FastAPI()

@app.get('/hello')
async def hello_world() -> dict:
    return {'message': 'Hello, world!'}

# tests/test_app.py
from app import app
from fastapi.testclient import TestClient

client = TestClient(app)

@pytest.mark.parametrize('method, path, status_code', [
    ('GET', '/hello', 200)
])
def test_hello_world(method, path, status_code):
    response = client.request(method, path)
    assert response.status_code == status_code
    assert response.json() == {'message': 'Hello, world!'}

Request Parameters and Headers

You can also test parameters and headers passed to your endpoints. For example, let's test a route that accepts a query parameter:

# test_app.py
from fastapi import FastAPI
import pytest

app = FastAPI()

@app.get('/items', response_model=list[dict])
async def get_items(q: str | None = None, skip: int = 0, limit: int = 10):
    return [{'item_name': f'item-{i}'} for i in range(skip, skip + limit)]

# tests/test_app.py
from app import app
from fastapi.testclient import TestClient

client = TestClient(app)

@pytest.mark.parametrize('method, path, q, skip, limit, status_code, response_data', [
    ('GET', '/items?q=foo', 'foo', 0, 10, 200, [{'item_name': 'item-0'}, {'item_name': 'item-10'}]),
    ('GET', '/items?q=foo&skip=1', 'foo', 1, 10, 200, [{'item_name': 'item-11'}, {'item_name': 'item-20'}]),
    ('GET', '/items?q=foo&skip=2', 'foo', 2, 10, 200, [{'item_name': 'item-12'}, {'item_name': 'item-21'}]),
])
def test_items(method, path, q, skip, limit, status_code, response_data):
    response = client.request(method, path, params={'q': q, 'skip': skip, 'limit': limit})
    assert response.status_code == status_code
    assert response.json() == response_data

Common Pitfalls

Mocking Dependencies

Often, your FastAPI app depends on external services or databases. When testing, you'll want to mock these dependencies to avoid making real calls. For example, if your app fetches data from a database, you can use a library like pytest-mock to mock the database calls.

# tests/test_app.py
from fastapi import FastAPI
from fastapi.testclient import TestClient
from pytest_mock import MockFixture

app = FastAPI()

def _dummy_db():
    return DummyDB()

class DummyDB:
    def fetch_data(self, id: int) -> dict:
        return {'id': id, 'name': f'item-{id}'}

app.dependency_overrides[_dummy_db] = _dummy_db

client = TestClient(app)

@pytest.mark.parametrize('method, path, id, status_code, response_data', [
    ('GET', '/items/{id}', 1, 200, {'id': 1, 'name': 'item-1'}),
    ('GET', '/items/{id}', 2, 200, {'id': 2, 'name': 'item-2'}),
])
def test_items(method, path, id, status_code, response_data):
    response = client.request(method, path, params={'id': id})
    assert response.status_code == status_code
    assert response.json() == response_data

Handling HTTP Errors

Sometimes, you'll want to test error scenarios, such as when a required field is missing. You can mock HTTP errors using pytest's pytest.raises fixture.

# tests/test_app.py
from fastapi import FastAPI, HTTPException
from fastapi.testclient import TestClient

app = FastAPI()

class DummyDB:
    def fetch_data(self, id: int) -> dict:
        return {'id': id, 'name': f'item-{id}'}

app.dependency_overrides[_dummy_db] = _dummy_db

client = TestClient(app)

@app.get('/items/{id}', response_model=dict)
async def get_item(id: int):
    return await _dummy_db.fetch_data(id)

# tests/test_app.py
from app import app

@pytest.mark.parametrize('method, path, id, status_code', [
    ('GET', '/items/100', 100, 400)
])
def test_get_item_with_invalid_id(method, path, id, status_code):
    response = client.request(method, path, params={'id': id})
    assert response.status_code == status_code
    assert response.json() == {'detail': 'Invalid id: 100' }

Conclusion

Testing FastAPI endpoints is an essential part of building reliable and maintainable applications. By setting up proper tests, you can ensure that your API behaves as expected. Remember to mock dependencies and handle error scenarios to get the most out of your tests. Happy testing!

Practical Introduction to Vector Databases

Dhaarani Dhanasekeran — Wed, 04 Nov 2026 14:33:13 +0000

Introduction

Vector databases are powerful tools for storing, querying, and retrieving high-dimensional data, such as text and image embeddings. In this article, we'll introduce vector databases from a practical perspective and explore their use cases. We'll cover their structure, how they store data, and discuss some common examples of their application.

1. What are Vector Databases?

Vector databases store data as vectors, which are high-dimensional vectors representing entities like text or images. Instead of storing the original data (e.g., text strings, image pixel values), vector databases store vectors that encode the semantic meaning of the data. This allows for efficient similarity searches and retrieval.

2. Structure and How They Work

Vector databases typically consist of a storage layer and a search layer. The storage layer stores the vectors and metadata. The search layer handles similarity queries, enabling efficient retrieval of relevant data. Vector databases can be either distributed or on-premises, and they often support various query languages or APIs.

3. Practical Examples and Code

Let's dive into a concrete example using Python and a vector database library called Chroma. Chroma is a popular library for working with vector databases and comes with a simple Python API for interacting with vector databases.

First, install Chroma and its storage backends:

pip install chromadb[pinecone]

Next, create a vector database collection and add some embeddings:

from chromadb.api.types import CollectionMetadata, EmbeddingFunction, MetadataProperty
from chromadb.db import get_db
from chromadb.api.models.embedding_function import EmbeddingFunctionModel

# Initialize a database client
client = get_db()

collection_metadata = CollectionMetadata(name='example_collection', metadata={})
embedding_function = EmbeddingFunctionModel(implementation=EmbeddingFunction.PY_FUNCTION, parameters={})
client.create_collection(collection_metadata, embedding_function)

# Add some embeddings
from chromadb.utils import embedding_functions

embeddings = embedding_functions.text_embedding_function('test.txt')
client.add(\'example_collection\
', embeddings, ids=['id1', 'id2'])

This creates a collection named 'example_collection' and adds two embeddings with IDs 'id1' and 'id2'.

4. Common Pitfalls and Considerations

When working with vector databases, several considerations are important. First, choose a suitable storage backend based on your use case (e.g., Pinecone for distributed databases, local storage for on-premises setups). Second, consider the trade-offs between storage efficiency and query performance. Third, ensure data privacy and compliance with regulations such as GDPR when storing sensitive data.

5. Conclusion

Vector databases are a valuable tool for developers looking to efficiently store and search for high-dimensional data. By understanding their structure and limitations, developers can leverage vector databases to build smarter and more personalized applications. Whether for text search, image retrieval, or other applications, vector databases offer a flexible and scalable solution.

6. Additional Resources

Chroma documentation: https://www.chromadb.org/docs/.
Pinecone documentation: https://pinecone.io/docs/.
Text embedding functions: https://www.chromadb.org/docs/plugins/embeddings/.