DEV Community: Timilehin Olusegun

EF Core Query Optimization: A Real-World Case Study

Timilehin Olusegun — Mon, 07 Jul 2025 05:30:00 +0000

When our team imported about 90,000 historical records into our production database, we discovered that our Entity Framework Core queries, which had performed well with smaller datasets, suddenly began timing out. This case study examines the performance bottlenecks we encountered and the optimization techniques that resolved them.

The Context

Our application had been running smoothly with a modest dataset of about 1,000 records. The dashboard loaded consistently in under 2 seconds, monitoring showed healthy performance metrics, and users were satisfied with the response times.

The data migration was intended to be routine—importing historical records from a legacy system to provide users with years of valuable information.

However, when users began accessing the system, we encountered significant performance issues.

What Went Wrong

The data import exposed fundamental scaling issues in our EF Core queries. What worked with 1,000 records failed completely with 90,000 records.

The Problem Manifests

The frontend team reported that API endpoints were returning 500 errors. Our investigation revealed:

User Impact:

Frontend applications receiving 500 errors from API endpoints
Dashboard pages failing to load with error messages
API requests timing out after 30 seconds
Previously fast-loading data lists now completely failing

Technical Symptoms:

Application logs filled with database timeout exceptions
Database connection pool reaching capacity
Server CPU utilization spiking during peak usage
Previously reliable endpoints failing consistently

Initial Troubleshooting

Our first step was examining the application logs, which revealed a clear pattern:

Microsoft.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired.
The timeout period elapsed prior to completion of the operation or the server is not responding.
 ---> System.ComponentModel.Win32Exception (258): The wait operation timed out.
 ...

These timeout errors appeared across multiple endpoints, but all shared a common characteristic: they involved querying our primary data table, which had grown from 1,000 to about 90,000 records.

The correlation was unmistakable—every failing operation was related to queries against the expanded dataset.

The Scaling Challenge

This scenario illustrates a common scaling challenge in application development. Code that performs adequately with small datasets can become unusable when data volume increases significantly. This wasn't a defect in our implementation—it was a fundamental scalability limitation.

Why the increase in data volume caused complete failure:

Query Complexity Scaling: Operations that appear linear often have hidden exponential costs
Database Execution Plan Changes: Query optimizers choose different strategies for larger datasets
Memory Allocation Issues: Loading complete result sets becomes problematic at scale
Index Efficiency: Indexing strategies effective for small tables may not scale appropriately

The application was functioning exactly as designed—the issue was that our design assumptions no longer matched our operational reality.

Identifying the EF Core Bottlenecks

With the correlation established between our data volume increase and the timeout errors, we needed to understand what was happening at the database level. The next step was enabling Entity Framework Core's query logging to see exactly what SQL was being generated.

Enabling EF Core Query Logging

First, we added detailed logging to our application configuration:

// In Program.cs or Startup.cs
builder.Services.AddDbContext<ApplicationDbContext>(options =>
{
    options.UseSqlServer(connectionString);
    options.EnableSensitiveDataLogging(); // For development only
    options.LogTo(Console.WriteLine, LogLevel.Information);
});

The Include() Performance Trap

We discovered that our dashboard queries were using Entity Framework's .Include() method to load related data. This seemed reasonable for our original small dataset:

// The problematic code that worked fine with 1,000 records
public async Task<List<InfrastructureItem>> GetDashboardDataAsync(string status)
{
    return await _context.Infrastructures
        .Include(i => i.LocalGovernmentArea)
        .Include(i => i.Region)
        .Include(i => i.SenatorialDistrict)
        .Include(i => i.AssessedBy)
        .Include(i => i.CreatedBy)
        .Where(i => i.Status == status)
        .ToListAsync();
}

What this generated in SQL:

SELECT [i].[Id], [i].[Name], [i].[Status], [i].[CreatedAt],
       [l].[Id], [l].[Name], [l].[Code],
       [r].[Id], [r].[Name], [r].[Code],
       [s].[Id], [s].[Name], [s].[Code],
       [u].[Id], [u].[UserName], [u].[Email],
       [u2].[Id], [u2].[UserName], [u2].[Email]
FROM [Infrastructures] AS [i]
LEFT JOIN [LocalGovernmentAreas] AS [l] ON [i].[LocalGovernmentAreaId] = [l].[Id]
LEFT JOIN [Regions] AS [r] ON [i].[RegionId] = [r].[Id]
LEFT JOIN [SenatorialDistricts] AS [s] ON [i].[SenatorialDistrictId] = [s].[Id]
LEFT JOIN [AspNetUsers] AS [u] ON [i].[AssessedById] = [u].[Id]
LEFT JOIN [AspNetUsers] AS [u2] ON [i].[CreatedById] = [u2].[Id]
WHERE [i].[Status] = @status

The Hidden Cost

With 90,000 records, this single query was:

Joining 6 tables for every row returned
Loading unnecessary data — we only needed names, not full user profiles
Transferring massive amounts of data from database to application
Consuming excessive memory to hold all the joined data

The database was working exponentially harder, and the network was transferring far more data than needed.

The Optimization Solution

We replaced the expensive joins with a more efficient approach:

// Optimized version - get only what we need
public async Task<List<InfrastructureItem>> GetDashboardDataAsync(string status)
{
    // Step 1: Get core infrastructure data only
    var infrastructureData = await _context.Infrastructures
        .AsNoTracking() // Important: no change tracking needed
        .Where(i => i.Status == status)
        .Select(i => new
        {
            i.Id,
            i.Name,
            i.Status,
            i.CreatedAt,
            i.LocalGovernmentAreaId,
            i.AssessedById,
            i.CreatedById
        })
        .ToListAsync();

    // Step 2: Get lookup data efficiently
    var lgaIds = infrastructureData
        .Where(i => i.LocalGovernmentAreaId.HasValue)
        .Select(i => i.LocalGovernmentAreaId.Value)
        .Distinct()
        .ToList();

    var lgaLookup = await _context.LocalGovernmentAreas
        .Where(lga => lgaIds.Contains(lga.Id))
        .ToDictionaryAsync(lga => lga.Id, lga => lga.Name);

    // Step 3: Map to final result
    return infrastructureData.Select(i => new InfrastructureItem
    {
        Id = i.Id,
        Name = i.Name,
        Status = i.Status,
        CreatedAt = i.CreatedAt,
        LocalGovernmentArea = i.LocalGovernmentAreaId.HasValue 
            ? lgaLookup.GetValueOrDefault(i.LocalGovernmentAreaId.Value, "Unknown")
            : "Unknown"
    }).ToList();
}

Immediate Results

The optimization had an immediate and dramatic impact on our most problematic endpoints. Most importantly, our users could reaccess their dashboards without encountering timeout errors.

Key Lessons Learned

.Include() doesn't scale linearly - each join multiplies the complexity
.AsNoTracking() is crucial for read-only operations
Projection with .Select() dramatically reduces data transfer
Separate queries can be faster than complex joins for large datasets

Conclusion

Our EF Core optimizations transformed query performance from 30+ second timeouts to 2-3 second responses. The key lesson: .Include() doesn't scale linearly with data volume.

Key takeaways:

Use .Select() for projection instead of .Include() for large datasets
Add .AsNoTracking() for read-only operations
Separate queries often outperform complex joins
Enable EF Core logging to identify bottlenecks

While this solved our immediate crisis, some queries still struggled with sorting and pagination on large datasets. The real breakthrough came with database indexing—the subject of Part 2.

Decoupling Audit Logs with RabbitMQ in .NET: An Intro to Event-Driven Architecture

Timilehin Olusegun — Fri, 07 Feb 2025 13:34:32 +0000

Introduction

Imagine you’re running a busy restaurant. Every time an order is placed, the chef has to stop cooking, walk over to the counter to take the order in a notebook, and then return to the kitchen. That sounds inefficient, right? Now, what if we had a waiter who simply relayed the order to the chef while they continued cooking? That’s essentially what event-driven architecture does for applications—it allows different system parts to function independently, processing tasks in parallel without causing delays or dependencies.

This article explores how event-driven architecture (EDA) separates audit logging from core application logic for better performance and scalability. We’ll introduce RabbitMQ as our message broker and discuss how it enables scalable, reliable logging in a .NET Web API. By the end of this article, you’ll understand why EDA is useful for audit logging and follow a step-by-step guide to implementing it in your projects throughout this series.

What is Event-Driven Architecture?

Event-driven architecture (EDA) is a design pattern where system components communicate by emitting and responding to events. Instead of direct interactions between services, events act as messages that different parts of the system can consume at their own pace.

Key Concepts of Event-Driven Architecture

Event Producer: The component that generates an event (e.g., a user action in a web app).
Event Broker: The intermediary that ensures events are delivered to consumers (e.g., RabbitMQ).
Event Consumer: The component that processes the event (e.g., storing audit logs in a database).

This approach decouples services, making applications more scalable, fault-tolerant, and easier to maintain.

Why Use Event-Driven Architecture for Audit Logging?

Traditional logging approaches often involve directly writing logs to a database when an action occurs. While this may work for small applications, it creates performance bottlenecks as traffic grows. Some challenges include:

Database Overhead: Frequent writes to the database can slow down the main application.
Tight Coupling: If logging is part of the main request cycle, failures in logging can affect the entire system.
Scalability Issues: Handling high volumes of logs can degrade performance over time.

By introducing event-driven logging, we separate the logging process from the main application workflow. The API simply publishes an event (e.g., "User logged in"), and a background service picks it up and stores it asynchronously. This ensures that even under heavy load, audit logs don’t interfere with the user experience.

Overview of the Architecture

We’ll use RabbitMQ as the message broker to decouple the audit logging process. Here’s a high-level view of our setup:

User performs an action (e.g., logs in, updates profile, makes a purchase).
The .NET Web API publishes an event to RabbitMQ.
RabbitMQ acts as a message broker, ensuring reliable delivery.
A background worker (event consumer) processes the event and stores it in a database.

This setup ensures that:

The main application stays responsive and continues handling requests smoothly.
Logging failures do not disrupt the user experience.
The system scales efficiently by adding more consumers as needed, improving throughput and reliability.

Choosing RabbitMQ for Message Queuing

RabbitMQ is a popular open-source message broker that facilitates communication between services. RabbitMQ is an excellent choice for audit logging because:

Reliability: Supports message acknowledgements and durability to prevent data loss.
Scalability: Can handle a large number of messages with multiple consumers.
Flexibility: Supports multiple messaging patterns like pub-sub and work queues.

While other message brokers like Kafka and Azure Service Bus exist, RabbitMQ strikes a good balance between ease of use and reliability for most .NET applications.

Next Steps

Now that we’ve established why event-driven architecture is ideal for audit logging, the next step is setting up RabbitMQ and integrating it with a .NET Web API. In the next article, we’ll:

Install and configure RabbitMQ.
Set up a .NET Web API to connect to RabbitMQ.
Send and receive simple messages.

Stay tuned as we dive deeper into the implementation in this series! 🚀

Conclusion

Event-driven architecture offers a flexible and efficient way to manage system interactions, making applications more scalable, resilient, and easier to maintain. By leveraging RabbitMQ as a message broker, we can ensure seamless and efficient log processing while keeping the core application responsive.

Whether you're dealing with high-volume logging or simply looking to improve system performance, this series will guide you through building a more efficient and reliable solution. Have you used event-driven architecture before? Drop your thoughts in the comments below! 🎉

Laravel Performance Tuning: Optimizing Database Queries for Scalability

Timilehin Olusegun — Wed, 29 Jan 2025 21:09:33 +0000

Ever worked on a Laravel project where database queries slowed down as traffic increased? I faced this recently while optimizing a real estate platform's backend, and I'll share some lessons I learnt.

Database optimization is one of the key areas in developing scalable and performant applications. Database optimization improves data retrieval, which leads to faster response times and lesser page load time. It can also help to reduce server load, thereby maximizing cost savings.

The Real Estate Platform Challenge

Picture this: You've built a beautiful real estate platform that serves multiple cities with advanced search filters. The property listings load quickly, search filters respond instantly, and everything seems perfect. Then, as your application scales and the user base grows, those queries that performed perfectly in development start taking longer and longer to execute. Sound familiar?

That's exactly what happened with our platform. Our wake-up call came through Sentry alerts flagging slow database queries in our production environment. The monitoring revealed search result queries taking upwards of 5 seconds to complete—not exactly the snappy experience we'd promised!

Common Query Pitfalls (And How to Avoid Them)

1. The N+1 Query Problem: Your Database's Secret Enemy

Remember playing those video games where defeating one enemy spawns multiple smaller ones? That's essentially what the N+1 query problem is like in Laravel. You fetch a list of properties, and then for each property, you make additional queries to fetch related data. Before you know it, your database is handling hundreds of queries instead of just one.

Here's what it typically looks like:

// Before optimization
$properties = Property::all();
foreach ($properties as $property) {
    echo $property->agent->name;  // This triggers a new query for each property
}

// After optimization
// Eager Loading with `with()`
$properties = Property::with(['agent'])->get();
foreach ($properties as $property) {
    echo $property->agent->name;  // No additional queries needed!
}

2. The Art of Database Indexing

Think of database indexes like the index of a book—they help you find what you're looking for without scanning every page. But there's much more to indexing than just throwing an index on every column. Let's dive deeper.

Understanding Different Types of Indexes

// Basic single-column index
Schema::table('properties', function (Blueprint $table) {
    $table->index('price');
});

// Composite index for multiple columns
Schema::table('properties', function (Blueprint $table) {
    $table->index(['city', 'price']); // Order matters!
});

// Unique index
Schema::table('properties', function (Blueprint $table) {
    $table->unique('property_code');
});

Index Strategy Best Practices

Column Order in Composite Indexes Matters

   // Good: Matches query pattern
   $properties = Property::where('city', 'New York')
                        ->whereBetween('price', [200000, 500000])
                        ->get();

   // Index should match this pattern
   $table->index(['city', 'price']); // City first, then price

Selective Indexing
Not every column needs an index. Some of the guidelines to choosing what columns to index are:
- Index columns frequently used in WHERE clauses and ORDER BY statements
- Index foreign key columns
- Don't index columns with low selectivity (like boolean flags)
Monitoring Index Usage

   -- MySQL query to check index usage
   SELECT
       table_name,
       index_name,
       index_type,
       stat_name,
       stat_value
   FROM mysql.index_statistics
   WHERE table_name = 'properties';

3. Select What You Need, Not Everything

One of the most common mistakes I've seen (and made!) is using select * by default. It's like going grocery shopping and buying the entire store when you only need ingredients for one meal. Here's a better approach:

// Instead of
$properties = Property::all();

// Be specific
$properties = Property::select(['id', 'title', 'price', 'city'])
    ->where('status', 'active')
    ->get();

4. Chunk Processing for Large Datasets

When dealing with large datasets, processing everything in a single operation can overwhelm your system's resources and create bottlenecks. Instead, use Laravel's chunk method to process records in manageable batches:

Property::chunk(100, function ($properties) {
    foreach ($properties as $property) {
        // Process each property
    }
});

5. Caching Strategies That Actually Work

Caching is like having a good assistant who remembers everything. But like any assistant, it needs clear instructions.

// Cache frequently accessed data
$properties = Cache::remember('featured_properties', 3600, function () {
    return Property::with('images')
        ->where('featured', true)
        ->get();
});

Pro tip: Don't cache everything! Focus on:

Data that's frequently accessed
Data that's expensive to compute
Data that doesn't change often

Best Practices to Take Away

Monitor First, Optimize Later
Don't fall into the trap of premature optimization. Use Laravel's built-in query log or tools like Telescope to identify actual bottlenecks.
Think in Sets, Not Loops
Whenever you find yourself writing a foreach loop that queries the database, step back and ask if there's a way to handle it in a single query.
Cache Strategically
Not everything needs to be cached. Focus on frequently accessed, computationally expensive queries that don't require real-time accuracy.
Index Thoughtfully
Think of indexes as a book's table of contents—you want enough detail to find things quickly, but not so much that the table of contents becomes longer than the book itself.

Common Pitfalls to Avoid

Don't "eager load" relationships you don't need
Avoid running queries in loops (the sneaky N+1 problem in disguise)
Don't cache everything—sometimes the overhead of cache management outweighs the benefits
Be careful with orderBy on non-indexed columns in large datasets
Don't create indexes for columns that are rarely used in WHERE clauses
Avoid updating indexed columns frequently; each update requires index maintenance

Conclusion: It's a Journey, Not a Destination

Database optimization isn't a one-time task you can check off your list. It's more like tending a garden—regular maintenance and attention yield the best results. Start with these basics, monitor your application's performance, and keep refining your approach.

Remember, the goal isn't to implement every optimization technique you know. It's about finding the right balance between code maintainability and performance for your specific use case. Sometimes, a simple eager loading statement can do more for your application's performance than hours spent on complex optimization strategies.

What optimization challenges have you faced and/or solved in your Laravel projects? Let’s discuss in the comments!

Unlocking Cloud Potential: File Upload to Azure Blob Storage in ASP.NET Core Applications

Timilehin Olusegun — Sun, 21 Jan 2024 20:00:08 +0000

In web development, efficient data storage is critical to the success of any application. One crucial aspect is handling file uploads, and cloud storage solutions like Azure Blob Storage have become valuable tools for developers. This article will show you how to seamlessly integrate file uploads to Azure Blob Storage in an ASP.NET Core application, ensuring a scalable and reliable solution.

Prerequisites

Web browser
.NET 8 SDK
Visual Studio Code
Basic knowledge of the ASP.NET Core MVC framework
An Azure account with a storage account created

Getting the connection string

To obtain your connection string, open your storage account on Azure and click Access Keys under the Security + Networking section.

Copy the connection string and put it in your appsettings.json file.

{
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft.AspNetCore": "Warning"
    }
  },
  "AllowedHosts": "*",
  "BlobConnectionString": "<PASTE_YOUR_CONNECTION_STRING_HERE>",
  "BlobContainerName": "<PASTE_YOUR_BLOB_CONTAINER_NAME_HERE>"
}

Setting Up the ASP.NET Core Application

Start by creating a new ASP.NET Core application or using an existing one. To create a new MVC application, run this command:

dotnet new mvc -n AzureUpload

You will be using the Azure.Storage.Blobs NuGet package to simplify interactions with Azure Blob Storage. Install the package using the following command in the terminal or Package Manager Console:

dotnet add package Azure.Storage.Blobs

Interacting with Azure Blob Storage

In your controller, create a method called UploadFile(). In this method, you will implement the logic for uploading files to Azure Blob Storage. The UploadFile() method will accept the file you want to upload and the filename as parameters.

private async Task UploadFile(IFormFile file, string filename)
{
    ...
}

In the UploadFile() method, you will retrieve the connection string for connecting with your Azure Storage account and the container name you specified in your configuration file, appsettings.json.

string? connectionString = _configuration.GetValue<string>("BlobConnectionString");
string? containerName = _configuration.GetValue<string>("BlobContainerName");

After that, you will initialise a BlobContainerClient object. This object takes the connection string and container name as parameters.

BlobContainerClient blobContainerClient = new BlobContainerClient(connectionString, containerName);

Finally, you will upload the file to Azure Storage using the BlobContainerClient object. You will create a MemoryStream object to hold the file contents temporarily. The UploadBlobAsync() method uploads it to Azure Storage. The method accepts the filename (it could also be a file path) and the stream.

using (var stream = new MemoryStream())
{
    await file.CopyToAsync(stream);
    stream.Position = 0;
    await blobContainerClient.UploadBlobAsync($"azure-upload/{filename}", stream);
}

In the end, your controller should look like this:

using System.Diagnostics;
using Microsoft.AspNetCore.Mvc;
using AzureUpload.Models;
using AzureUpload.Models.FileUploadViewModel;
using Azure.Storage.Blobs;

namespace AzureUpload.Controllers;

public class HomeController : Controller
{
    private readonly IConfiguration _configuration;

    public HomeController(IConfiguration configuration)
    {
        _configuration = configuration;
    }

    public IActionResult Index()
    {
        return View();
    }

    [HttpPost]
    public async Task<IActionResult> Upload(FileUploadViewModel fileUploadViewModel)
    {
        if (ModelState.IsValid)
        {
            try
            {
                await UploadFile(fileUploadViewModel.File, fileUploadViewModel.File.FileName);
            }
            catch(Exception e){
TempData["Error"] = $"An error occurred: {e.Message}. Please try again."
                return View();
            }
        }
        return View("Index", fileUploadViewModel);
    }

    private async Task UploadFile(IFormFile file, string filename)
    {
        string? connectionString = _configuration.GetValue<string>("BlobConnectionString");
        string? containerName = _configuration.GetValue<string>("BlobContainerName");
        BlobContainerClient blobContainerClient = new BlobContainerClient(connectionString, containerName);

        using (var stream = new MemoryStream())
        {
            await file.CopyToAsync(stream);
            stream.Position = 0;
            await blobContainerClient.UploadBlobAsync($"azure-upload/{filename}", stream);
        }
    }
}

Conclusion

As technology evolves, embracing cloud solutions like Azure becomes increasingly essential for building robust and scalable applications. With the Azure Storage Blobs package, integrating file uploads with Azure in an ASP.NET Core application is straightforward. You can effortlessly manage and scale your applications with the power of Azure, ensuring a reliable and efficient solution for handling file storage needs.

In this article, I have shown you how to upload files to Azure Storage in an ASP.NET Core application. You can get the example code discussed on GitHub. Cheers!

Visualizing Data in the Terminal: A Simple Guide to Building a Customized Data Visualization Tool

Timilehin Olusegun — Wed, 15 Mar 2023 11:20:39 +0000

As technology advances so does the amount of data generated and collected. With the rise of APIs, it has become easier than ever to access vast amounts of data. However, analysing and interpreting this data can be challenging without the right tools. In this article, we will explore how to build a program that can visualize data from an API in the terminal. This program can be a useful tool for anyone who needs to quickly analyze and interpret data without using complex visualization software.

Prerequisites

Basic programming skills. Python knowledge is a nice-to-have because that is the language we will be using.
Python 3.8 and above installed on your local machine.
Access to the terminal or command line as the program will be executed on the terminal or command line.

Building our Data Visualization Tool

Imagine we want to track the growth of the user base of a new product over time and analyze usage patterns to inform product decisions. We have access to an API endpoint that returns the number of users on specific dates, but we need a way to quickly and easily visualize this data. The program will:

Accept the time boundaries (i.e, start date and end date)
Fetch data from an API based on the start and end date.
Visualize the data as a graph on the terminal.

The first thing we will do is create a folder for the project. Let's name this folder visualize. Inside the folder, create a file called visualize.py. Then we cd into the folder and create a virtual environment. The virtual environment is an isolated space that enables us to install and manage Python modules and packages required for the project. So instead of installing packages globally, we can install them just for a specific project.

To create a virtual environment, run:



cd visualize
python3 -m venv env

Then we need to activate the virtual environment.
On Linux:



source ./env/bin/activate

On Windows:



env\Scripts\activate.bat

Accepting and parsing input from the terminal

To collect input from the terminal, we can use flags and arguments. Flags are also known as options, and they usually start with a dash (-). So to run our program when we are done, we would run a command like this:



python3 visualize.py -s <START DATE> -e <END DATE>

The -s and -e flags specify the START DATE and END DATE arguments respectfully. To parse flags and arguments from the terminal, we need to install a python module named argparse. To install, run:



pip install argparse

Then, in our visualize.py file, we import argparse and write a function to parse input from the terminal.



import sys
import argparse

def parse_arguments(args):
    try:
        parser = argparse.ArgumentParser(
            description='Visualize Data within a time frame', prog='visualize')
        parser.add_argument('-s', '--start-date', required=True,
                            help='Start date. Date format: DD-MM-YYYY', type=date)
        parser.add_argument('-e', '--end-date', required=True,
                            help='End date. Date format: DD-MM-YYYY', type=date)

        parsed_arg = parser.parse_args(args)
        return parsed_arg
    except argparse.ArgumentError as err:
        sys.exit(err)
    except argparse.ArgumentTypeError as err:
        sys.exit(err)


def date(date_string):
    return datetime.strptime(date_string, '%d-%m-%Y').date()

In the code snippet above, we create the ArgumentParser object. We can then specify a description text with the description parameter. The prog parameter specifies the name of the program. Next, we add arguments that we expect to be supplied when running the program. In our program, we need the start_date and the end_date arguments. So we can add arguments using the add_argument() method. In this method, we provide the flag that specifies the argument in both short and long format, for example, -s and --start-date for the start_date argument. We also specify if the argument is required or not.

We then have to specify the type of the arguments, which in our case, is the date type. We write a function that parses a date string into python date format and returns the date format. Finally, we add an explanation of the argument which will be displayed if we run the program incorrectly. We do all these for both the start_date argument and the end_date arguments. Then we parse the arguments using the parse_args() method. Finally, we return the parsed arguments.

Getting coordinates for plotting the graph

Next, we want to get all our coordinates for plotting our graph. We create a function that accepts the parsed arguments as a parameter and then returns our x and y coordinates.



def get_coordinates(args):
    x_values = []
    y_values = []


    api_data = {
        "01-01-2022": 300,
        "02-01-2022": 500,
        "03-01-2022": 700,
        "04-01-2022": 1300,
        "05-01-2022": 2000,
        "06-01-2022": 3000,
        "07-01-2022": 3500,
        "08-01-2022": 4000,
        "09-01-2022": 4500,
        "10-01-2022": 5000,
        "11-01-2022": 20000,
        "12-01-2022": 35000,
        "13-01-2022": 46000,
        "14-01-2022": 70000,
        "15-01-2022": 90000
    }
    start_date = args.start_date
    end_date = args.end_date

    if start_date.strftime('%d-%m-%Y') in api_data and end_date.strftime('%d-%m-%Y') in api_data:
        for key, value in api_data.items():
            if date(key) >= start_date and date(key) <= end_date:
                x_values.append(date(key))
                y_values.append(value)
    elif start_date.strftime('%d-%m-%Y') not in api_data:
        sys.exit("Given start date is not available")
    elif end_date.strftime('%d-%m-%Y') not in api_data:
        sys.exit("Given end date is not available")
    else:
        sys.exit("An error occurred. Please check your input")

    coordinates = {
        "X": x_values,
        "Y": y_values
    }
    return coordinates

First, we have our API data. In the get_coordinates() function, create the x_values and y_values lists (also known as arrays in other languages) that will hold values of our x-axis and y-axis. We then initialize a variable, api_data, which is a dictionary (also known as Maps or HashMaps in other languages ) to hold mock API data for testing purposes. In a production environment, this variable would be replaced with actual data retrieved from an API.

We then check if the start and end dates provided are available in the api_data. If they are available, we loop through the api_data dictionary and add the dates and values that fall within the range of the start and end dates specified in the arguments to the x_values and y_values lists, respectively.

In the loop, we use the date() function created earlier to parse date strings in our API data. The x_values and y_values lists represent the dates and number of users, respectively. If the start date or end date is not available in the res dictionary, the script exits with an appropriate error message. Then, we return the coordinates as a dictionary.

Plotting the graph

To plot the graph, we will use a python package called plotext. Plotext lets you plot scatter, line, bar, histogram, and date-time plots (including candlesticks) directly on the terminal.
First, we need to install this package.



pip install plotext

Then we import plotext in our code and write a function that takes the coordinates and plots a bar chart directly in the terminal using plotext.



import plotext as plt

def visualize_data(data):
    # data = get_data(args)

    plt.title("User Base Growth Graph")
    plt.ylabel("Number of Users")
    plt.xlabel("Dates")
    plt.date_form('d/m/Y')

    no_of_users = data.get("Y")
    dates = plt.datetimes_to_string(data.get("X"))


    plt.yticks(no_of_users)
    plt.bar(dates, no_of_users)

    plt.show()

In the visualize_data function, the labels for our y-axis and x-axis are set using the xlabel() method. We also specify the format to display our dates. Then we get the values of our y and x-axis as the variables, dates, and no_of_users. We need to convert our x values (date objects) to strings. Finally, we can plot our bar chart with the bar() function, passing in x and y values as dates and no_of_users.

Running our program

At the end of our program, we need to call the functions we created.



args =  parse_arguments(sys.argv[1:])
coordinates = get_coordinates(args)
visualize_data(coordinates)

The arguments are first gotten from the terminal using the python sys module and then parsed in the parse_arguments() function. The parsed arguments are then passed into the get_coordinates function. Finally, the visualize_data gets the coordinates and visualizes the data.
To run this program, we will cd into our project folder and run this command.



python -m visualize -s <START DATE> -e <END DATE>

Replace <START DATE> and <END DATE> with their respective values.
The output on our terminal will look like this:

Or like this, depending on your terminal:

Conclusion

Data visualization in the terminal can be a powerful tool for analyzing and interpreting data without the need for complex visualization software. By using Python and libraries such as Plotext, you can quickly and easily create powerful visualizations that help you make data-driven decisions. Whether you're tracking user growth or analyzing sales data, a terminal-based visualization tool can help you streamline your analysis and make sense of complex data.

In this article, we have demonstrated how to build a Python program to visualize data directly on the terminal. We handled input from the terminal using the argparse library, then processed and transformed both the parsed terminal input and API data for plotting. We then plotted a bar chart using the plotext package.

You can get the example code discussed on GitHub. Cheers!

Enhance Team Collaboration with Laravel Sail Share Command

Timilehin Olusegun — Thu, 02 Mar 2023 01:56:32 +0000

Have you ever found yourself in a situation where you've completed a local project, but your team is waiting for it to be deployed to the staging server before they can start working on it? That's what happened to me when I added a new feature to an API at work. I created a pull request, but it took too long to be approved and merged. That's when my teammate asked me to "share it." I was taken aback because I had no idea what he meant. But little did I know that sharing local projects online was much easier than I thought.

In this article, I'm going to guide you through sharing your local Laravel projects online for anyone to access, just like my teammate showed me.

Prerequisites

Before we dive into the steps to share your local Laravel project, there are some conditions to be met:

A basic understanding of the Laravel framework and its components.
Installed Laravel Sail: To use the Laravel Sail Share Command, the reader should have Laravel Sail installed on their Laravel project. You can follow the instructions on the Laravel Sail documentation page to install Laravel Sail on your machine.
Docker: You need to have Docker installed on your local machine.
A Local Laravel Project that you want to share. Access to the terminal or command line as the Laravel Sail Share Command is executed through the terminal or command line.

An idea of containerisation and how Docker works is a good nice-to-have, but it’s not compulsory for this article. These prerequisites will ensure you have the necessary setup to follow along with the steps outlined in the article.

What is Laravel Sail?

Laravel Sail is a simple command-line interface that gives you access to Laravel's built-in Docker development environment. It is a great way to set up and start building your Laravel application using PHP, MySQL, and Redis without prior Docker experience. With Laravel Sail, developers can set up their Laravel development environment quickly and easily without worrying about the underlying infrastructure and dependencies.

Additionally, Laravel Sail provides some useful commands for managing your Laravel project, such as starting and stopping the development environment, running database migrations, and sharing your project online with others.

Using the Laravel Sail Share Command

First, you must launch your project by running the sail up command. This command will start all the Docker containers defined in your application's docker-compose.yml file (so you need to have Docker running on your machine). You can also run the sail up -d to start the containers in the background.

Once the containers are up and running, you can share your local Laravel project using the Laravel Sail Share command. This command to share your local Laravel project is



sail share

This command will create a publicly accessible URL that your team can use to access your local Laravel project. You will get something similar to the image shown below.

Now you can send the URL to the members of your team. The generated URL will allow them to access your local Laravel project anywhere, at any time.

Please note that to share your project using the share command, you must configure the trusted proxies within the TrustProxies middleware. It helps URL generation helpers like URL and route to figure out which HTTP host to use for URL generation.



    /**
     * The trusted proxies for this application.
     *
     * @var array|string|null
     */
    protected $proxies = "*";

The share command uses an open-source tunnelling service called Expose. Expose is built by BeyondCode. The sail share command runs Expose under the hood.

Conclusion

Thanks to the Laravel Sail share command, sharing your local Laravel project with others is easy. It can help to demonstrate your work, get feedback, or collaborate with others. Just remember to stop sharing your project by using the sail down command.

A different situation where sharing your local project can be helpful is if you are developing a mobile application and want to test it with the API on your local machine. Personally, having this knowledge would have made my life easier years ago.

In this article, we have looked at how to share your local Laravel projects with your team using the Laravel Sail Share Command. Using these steps, you can enhance your team’s collaboration and make it easy to access your local Laravel projects. With the Laravel Sail Share Command, you can improve team communication and make sure that your team can function more effectively and efficiently.

Strategy Design Pattern in Python

Timilehin Olusegun — Sun, 05 Feb 2023 13:13:52 +0000

When developing software applications, depending on the type of application, there are questions to be answered in the design phase of development before writing actual code. One of these questions is, “How do I structure my application?”. Software Design Patterns can help to answer this question of structure. Design patterns help us structure our application in ways that make it robust, maintainable, and easy to understand.

Design patterns are high-level concepts to fix issues relating to structure in a codebase. They are not language specific, so you can use them in any language that supports Object Oriented Programming. In this article, we’ll learn about a popular design pattern called the Strategy Pattern.

Prerequisites

Before we go into strategy pattern, you need to have a basic understanding of Object Oriented Programming. Classes and objects are a core part of the design pattern concept.

What Is The Strategy Pattern?

Strategy Pattern is a design pattern that defines a collection of algorithms and allows these algorithms to be interchangeable at runtime. According to the book on design patterns, “Head First: Design Patterns” by Eric Freeman & Elisabeth Robson,

The Strategy Pattern defines a family of algorithms, encapsulates each one, and makes them interchangeable. Strategy lets the algorithm vary independently from clients that use it.

There are scenarios where you can have more than one way of achieving a particular task. These different ways are the algorithms. Traditionally, you tend to think of conditional statements (if-else) as a way of determining which algorithm to use. This method is not a better way to write good and clean code. Also, as the number of algorithms increase, the code becomes longer and harder to maintain.

In cases like this, the strategy pattern is a good solution. The strategy pattern allows you to define classes, called strategies, that contain the implementations of your various algorithms. The main class has a reference to the strategy and you can then pass your preferred strategy to the context. Let's look at an example implementation of the strategy pattern.

Implementation

Let’s say we have an e-commerce application where products are registered. When registering these products, we want to generate unique IDs for each of them. We want to be able to generate IDs using different criteria and parameters. We will use the strategy design pattern to develop two switchable strategies for generating the product ID. First, we’ll have a RandomStrategy that generates the product ID as a random string. In contrast, another DerivedStrategy generates the product ID using particulars of the product like the date-time, category, and product SKU.

We have our strategy.py file. This file contains the family of algorithms (in our case, strategies) for generating the product ID and their different implementations.



from abc import ABC, abstractmethod
from datetime import datetime
import string
import secrets


class ProductIdStrategy(ABC):
    """An interface for strategy type"""

    @abstractmethod
    def generate_product_id() -> None:
        """Each class will have its own implementation of this function"""
        pass


class RandomStrategy(ProductIdStrategy):

    def generate_product_id(self) -> str:
        limit = 12
        product_id = "".join(secrets.choice(
            string.ascii_uppercase+string.digits) for i in range(limit))
        return product_id


class DerivedStrategy(ProductIdStrategy):
    ''' The Derived Strategy will derive the product id from the id, category and SKU of the product'''

    def __init__(self, product) -> None:
        super().__init__()
        self.product = product

    def generate_product_id(self) -> str:
        id = self.product["id"]
        # Get the first 3 characters of the category
        category = self.product["category"][:3].upper()
        sku = self.product["sku"]
        # Get the date string and replace remove the hyphens
        date = datetime.strptime(self.product["date_added"], "%Y-%m-%d").date().__str__().replace("-", "")
        product_id = "".join([category, "-", date, "-", sku, id,])
        return product_id

Then we have a product.py file. In this file, we implement the different strategies interchangeably.



from strategy import *

class Product:
    _strategy: ProductIdStrategy

    def __init__(self, strategy: ProductIdStrategy) -> None:
        self._strategy = strategy

    def get_product_id(self) -> str:
        return self._strategy.generate_product_id()

if __name__ == "__main__":
    stock = {
        "id": "1",
        "name": "Maxi Lotion",
        "category": "Skin Care",
        "sku": "6134752",
        "date_added": "2022-12-28",
    }

    ''' Generating the product id using the random strategy '''
    strategy = RandomStrategy()
    product = Product(strategy)
    print("The product id using the Random Strategy is : " + product.get_product_id())

    '''
        Generating the product id using the derived strategy
        The product is passed into to the strategy so as to extract information from it
    '''
    strategy = DerivedStrategy(stock)
    product = Product(strategy)
    print("The product id using the Derived Strategy is : " + product.get_product_id())

The result shows two product IDs. One was generated using the Random Strategy and the other was generated with the Derived Strategy.

Conclusion

In this article, we have learned what the strategy pattern is and how to implement it. With the strategy design pattern, we can switch between algorithms at runtime without changing the code. The Strategy pattern may be used to select the application's behaviour instead of using conditional expressions.

I hope this article has shown you how to implement the strategy design pattern. You can get the example code discussed in this GitHub repository.

Handling Exceptions in Laravel: A Cleaner Method

Timilehin Olusegun — Wed, 14 Dec 2022 15:57:12 +0000

When building web applications, you will likely use various APIs and services to add functionality to your software. While using these external services, you might run into unexpected errors that cause the application to crash. To avoid these unexpected crashes and ensure the smooth running of your application, you need to handle them properly.

The conventional way of handling exceptions in the PHP programming language is by using try-catch blocks and the same method applies to Laravel. In this article, I will show you a shorter and cleaner way to handle exceptions in Laravel.

In this article, I will be showing you a shorter and cleaner way to handle exceptions in Laravel.

Prerequisites

Before you begin this tutorial you'll need the following:

Web Browser
Your preferred IDE
Basic knowledge of PHP and the Laravel framework

What are Exceptions?

Exceptions are occurrences that emerge while a program is being executed and causes a disruption in the execution flow of the program. When an exception occurs, the program crashes and the user cannot continue using the program. The program will have to be restarted for it to be used again. In order to avoid these occurrences, we have to handle them.

Handling Exceptions in Laravel

Now we have a simple form with an input field which allows us to search for a user by email and then display their details.

So we have two routes.



Route::get('/', [UserController::class, 'index']);
Route::post('/search', [UserController::class, 'search'])->name('search');

We also have your Blade template file, resources/views/index.blade.php where we have the code for the form.



<form action="{{ route('search') }}" method="post">
    @csrf
    <div class="mb-4 mt-4">
        <div class="form-floating mb-3">
            <input type="email" name="email" id="floatingEmail" class="form-control @error('email') is-invalid @enderror" value="{{ old('email') }}" placeholder="Enter your email">
            <label for="floatingEmail">Email address</label>
        </div>
        @error('email')
            <span class="text-danger mb-1">{{ $message }}</span>
        @enderror
    </div>
    <div class="d-grid col-6 col-md-3 mx-auto">
        <button class="btn btn-secondary" type="submit" id="store">Search</button>
    </div>
</form>

In the controller, we have two methods; the index method and the search method. The index method shows the index page. The search method validates the input email and then uses it to search for the user’s details in the database.



class UserController extends Controller
{
    public function index()
    {
        return view('index');
    }


    public function search(Request $request)
    {
        $validated = $this->validate($request, [
            'email' => 'required|email',
        ]);
        $email = $validated["email"];

        $user = User::where('email', $email)->first();
        return view('result', compact('user'));

    }

}

When we search for a user by email and the user is found, we get the user’s details as shown below.

Now, what happens when a user is not found?

Exception Handling using the Try-Catch block

If the user is not found, we get an error exception page.

This error page needs to help the user understand what is going on. We could use firstOrFail() instead of first(). This will make Laravel display a “404 | Not Found” page.

But this page is also not really helpful to the user. What we need to do is to catch the exception, and then display a message that is understandable.

In Laravel, firstOrFail() throws an Eloquent exception, ModelNotFoundException. With this knowledge, we can handle the exception using the try-catch block in the search method of the User Controller.



public function search(Request $request)
{
    $validated = $this->validate($request, [
        'email' => 'required|email',
    ]);

    $email = $validated["email"];

    try {
        $user = User::where('email', $email)->firstOrFail();
    } catch (ModelNotFoundException $exception) {
        return back()->with('error',$exception->getMessage())->withInput();
    }
}

Now we can show an error on the homepage. So we go to our Blade template file, resources/views/index.blade.php to display the error message.



@if (session('error'))
<div class="row justify-content-center">
    <div class="alert alert-danger alert-dismissible fade show col-md-6" role="alert">
        <strong>Message:</strong> {{ session('error') }}
        <button type="button" class="btn-close" data-bs-dismiss="alert" aria-label="Close"></button>
    </div>
</div>
@endif

This is the result of displaying the error message.

Now imagine we have to handle just more than one exception. We have to catch each exception one by one. Also when using external services in your application, these external services could have multiple custom exception classes depending on the error encountered. A good example is the Stripe library for Stripe Payments API.



try {
  // Use Stripe's library to make requests...
} catch(\Stripe\Exception\CardException $e) {
  // Since it's a decline, \Stripe\Exception\CardException will be caught
  echo 'Status is:' . $e->getHttpStatus() . '\n';
  echo 'Type is:' . $e->getError()->type . '\n';
  echo 'Code is:' . $e->getError()->code . '\n';
  // param is '' in this case
  echo 'Param is:' . $e->getError()->param . '\n';
  echo 'Message is:' . $e->getError()->message . '\n';
} catch (\Stripe\Exception\RateLimitException $e) {
  // Too many requests made to the API too quickly
} catch (\Stripe\Exception\InvalidRequestException $e) {
  // Invalid parameters were supplied to Stripe's API
} catch (\Stripe\Exception\AuthenticationException $e) {
  // Authentication with Stripe's API failed
  // (maybe you changed API keys recently)
} catch (\Stripe\Exception\ApiConnectionException $e) {
  // Network communication with Stripe failed
} catch (\Stripe\Exception\ApiErrorException $e) {
  // Display a very generic error to the user, and maybe send
  // yourself an email
} catch (Exception $e) {
  // Something else happened, completely unrelated to Stripe
}

As we can see, you have to catch every single error so as to handle them properly to suite your application’s needs. This makes the lines of code we have to write increase and this can make our code harder to maintain. Though this method is perfectly fine, there is an even shorter and slicker way to handle exceptions with Laravel.

Handling Exceptions Using the Rescue Helper

Laravel provides us with the rescue helper which executes your code or task as a closure. The rescue helper catches any exceptions that occur during the execution of the closure. Any exception caught is sent to the exception handler in your Laravel application.



public function search(Request $request)
{
    $validated = $this->validate($request, [
        'email' => 'required|email',
    ]);

    $email = $validated["email"];

    return rescue(function () use ($email) {
        $user = User::where('email', $email)->firstOrFail();
        return view('result', compact('user'));
    }, function ($exception) {
        return back()->with('status', $exception->getMessage());
    }, true);
}

Generally, the rescue method can accept three arguments.

The first argument is a closure that contains the piece of code you want to execute. In our case, we want to search for the user by email and display their details on another page.
The second argument (optional) is the closure executed if an exception occurs in the previous argument. You can also pass in an argument representing the exception thrown.
The third argument (optional) accepts a boolean value. This argument tells the function whether or not to report the exception to the configured exception handler. If true, the rescue() helper reports the exception and vice-versa.

Conclusion

In cases where you might have multiple exceptions to handle, the rescue() helper can help you reduce the number of lines of code you have to write. It can also help improve the readability of your code thereby making it easier to maintain over time.

I hope this article has shown you how to handle exceptions more shortly and cleanly. You can get the example code discussed on GitHub. Cheers!

DEV Community: Timilehin Olusegun

EF Core Query Optimization: A Real-World Case Study

The Context

What Went Wrong

The Problem Manifests

Initial Troubleshooting

The Scaling Challenge

Identifying the EF Core Bottlenecks

Enabling EF Core Query Logging

The Include() Performance Trap

The Hidden Cost

The Optimization Solution

Immediate Results

Key Lessons Learned

Conclusion

Decoupling Audit Logs with RabbitMQ in .NET: An Intro to Event-Driven Architecture

Introduction

What is Event-Driven Architecture?

Why Use Event-Driven Architecture for Audit Logging?

Overview of the Architecture

Choosing RabbitMQ for Message Queuing

Next Steps

Conclusion

Laravel Performance Tuning: Optimizing Database Queries for Scalability

The Real Estate Platform Challenge

Common Query Pitfalls (And How to Avoid Them)

1. The N+1 Query Problem: Your Database's Secret Enemy

2. The Art of Database Indexing

Understanding Different Types of Indexes

Index Strategy Best Practices

3. Select What You Need, Not Everything

4. Chunk Processing for Large Datasets

5. Caching Strategies That Actually Work

Best Practices to Take Away

Common Pitfalls to Avoid

Conclusion: It's a Journey, Not a Destination

Unlocking Cloud Potential: File Upload to Azure Blob Storage in ASP.NET Core Applications

Prerequisites

Getting the connection string

Setting Up the ASP.NET Core Application

Interacting with Azure Blob Storage

Conclusion

Visualizing Data in the Terminal: A Simple Guide to Building a Customized Data Visualization Tool

Prerequisites

Building our Data Visualization Tool

Accepting and parsing input from the terminal

Getting coordinates for plotting the graph

Plotting the graph

Running our program

Conclusion

Further Reading

Enhance Team Collaboration with Laravel Sail Share Command

Prerequisites

What is Laravel Sail?

Using the Laravel Sail Share Command

Conclusion

Further Reading

Strategy Design Pattern in Python

Prerequisites

What Is The Strategy Pattern?

Implementation

Conclusion

Handling Exceptions in Laravel: A Cleaner Method

Prerequisites

What are Exceptions?

Handling Exceptions in Laravel

Exception Handling using the Try-Catch block

Handling Exceptions Using the Rescue Helper

Conclusion

Further Reading