DEV Community: Erick Badillo

My Journey with Microsoft: From a Parallel Port to Azure and AI

Erick Badillo — Sat, 05 Apr 2025 01:50:05 +0000

Some memories never fade — they remain etched into your identity, quietly shaping your path. For me, many of those memories are tied to code, a keyboard, and a deep passion for building.

At just 16, I discovered the C language. The feeling of writing something that a machine could “understand” was nothing short of magical. From there, I quickly dove into C++, Visual Basic, and even COBOL — yes, COBOL — driven by an insatiable curiosity to understand how software worked. I spent every spare hour studying, experimenting, and tinkering, fascinated by the power of programming.

In university, that curiosity deepened. I added other languages and tools — Java, Linux environments, Prolog, to name a few. I remember several projects I worked on during that time, but two of them I recall with special care.

The first was a suite of console commands to program an 8-bit microcontroller through a parallel port. I built the entire thing over a single weekend, fully immersed in the challenge, using C and a Microsoft compiler. That small project helped me pass the course — but more importantly, it helped me to learn new things and brought me real joy. I loved the feeling of solving meaningful problems with code.

The second came during a system design course, when I picked up a worn-out book on Microsoft Visual Basic 6.0. I spent countless days and nights working alone, learning to build complete applications in the classic IDE. It was a simple inventory system for a music store to track CDs and vinyls, but it was my first real taste of building something with structure, purpose, and utility. I remember taking my first steps with databases and authentication, and even building an installer that ran from floppy disks — yes, actual floppy disks.

I still remember installing MS-DOS and Windows 3.1 using diskettes, and I genuinely enjoyed the process — carefully watching the screen, waiting for the message prompting the next disk, and swapping each one in order. Seeing my own installer work the same way was exciting. It made me feel like I was building something real — something that belonged in that world.

That project was the appetizer for what would eventually become a full-course journey into the world of enterprise applications, even though I had no idea of it at the time.

Lessons from the Field

Then came the opportunity that truly launched my professional journey: I began working at a bank, joining the corporate banking team as a junior developer. I was young and full of energy, eager to take on any challenge. The systems we worked on were large, complex, and critical to the business — and I found myself right in the middle of it all.

As the months passed, I kept growing. The scale and rigor of financial systems demanded more from me than anything I had experienced before. It pushed me to level up, fast. I was given room to learn, take responsibility, and little by little, I advanced — eventually becoming the technical lead of the team.

It was during this time that I worked with Classic ASP, and later transitioned to .NET 1.1, C#, and ASP.NET Web Forms. Those years shaped not just my skills, but my mindset as an engineer. I was no longer just writing code — I was learning to lead, to design, and to solve problems at scale.

And along the way, I had the privilege of learning from some of the best. I still remember working with a pair of Premier Field Support Engineers from Microsoft, who helped us troubleshoot critical issues in our banking platform. Our conversations over lunch or coffee breaks were mini-masterclasses. One of them especially inspired me — I watched in awe as he debugged our application by analyzing intermediate .NET code directly in the staging environment, without even needing the original source code. That moment blew my mind. It showed me what deep technical mastery could look like — and it’s something I’ve carried with me ever since.

A Partnership That Shaped My Career

Throughout all these years, Microsoft technologies have been a constant companion. I’ve seen platforms evolve, paradigms shift, and tools become more powerful and elegant — and with each new version of Visual Studio or .NET, I felt more at home.

Today, nearly 20 years later, I use Visual Studio Enterprise, Visual Studio Code, Windows 11, Office 365, OneDrive, GitHub, GitHub Copilot, and Azure every day — not only for self-learning, but for solving real-world problems in my projects. Most recently, I’ve been implementing Azure AI services in banking applications, and I’m still just as passionate as I was when I wrote my first main() function.

What Comes Next: Sharing the Journey

Inspired by this long and fulfilling journey, I’ve decided to take a new step forward: I’ll soon be launching a training initiative focused on advanced .NET development. My goal is to combine deep technical concepts with the insights I’ve gained over the years working in the industry — building not just systems, but resilient, scalable, and meaningful software.

Along the way, I’ll also share some of what I’ve learned in Artificial Intelligence, and how it’s reshaping the way we think about building applications. And for those just getting started, I’ll be offering tips, practical advice, and encouragement, drawing from the lessons of my own path.

Full Circle

From programming 8-bit microcontrollers through a parallel port, to implementing enterprise-scale AI solutions in the cloud — it’s been a long road filled with passion, long nights, breakthroughs, and a relentless desire to keep growing.

If you’ve ever felt that spark — that need to understand, to build, to push yourself forward — then this journey is for you too.

Stay tuned

RAG with Embeddings in .NET: Enhancing Semantic Search

Erick Badillo — Fri, 12 Apr 2024 23:54:24 +0000

In this post, I present you a solution developed within the .NET ecosystem that shows how to integrate RAG with the ETL Medallion architecture using the Azure AI Document services, Pinecone vector database, Azure Cosmos DB, and the chat and embeddings OpenAI API endpoints to create a system capable of providing good responses in the context of Mexican Tax legislation.

Theoretical Framework

Medallion Architecture

The Medallion architecture organizes data transformation into three distinct layers—bronze, silver, and gold—each optimized for specific roles in the data lifecycle from raw ingestion to business-ready "golden records."

Embeddings and Vector Databases

Embeddings are numeric representations of the text, capturing their semantic relationships. During the embedding process, a vector is created, and each vector is stored in Pinecone (vector database), This setup enables the system to efficiently pull contextually relevant data using semantic searches over the stored information.

Retrieval-Augmented Generation (RAG)

RAG combines the strengths of both retrieval-based and generative AI systems to enhance the model's response accuracy and relevance. By dynamically retrieving relevant document segments, RAG allows models to generate better responses to the user's needs.

Prompt Engineering

Prompt Engineering involves designing inputs that influence the behavior of LLMs. It is crucial for adapting LLM outputs to specific tasks or contexts, especially when direct training data is limited.

Few-Shot Learning Techniques
Few-shot learning involves introducing a minimal number of examples to a pre-trained model, enabling it to apply its generalized knowledge to new tasks. This technique is used in the project as part of the golden document creation.

Enhancing RAG with Contextual Prompts
For the (RAG) process, the system's effectiveness is improved by incorporating relevant information into the context window. This method involves crafting prompts that include specific contextual information, which guides the retrieval mechanism to produce precise and relevant outputs.

Hypothesis.

Enriching a vector database with synthetic or human-generated questions related to document content, along with structured summarizations (chunks) of these documents, significantly enhances the retrieval process in a Retrieval-Augmented Generation (RAG) system.

Synthetic or Human-Generated Questions:
These anticipate user queries, improving the retrieval model’s ability to match actual user questions with relevant stored queries, resulting in more accurate and rapid information retrieval.

Structured Summarizations:
Condensing content into coherent segments reduces computational load and focuses the retrieval on relevant text areas, speeding up the process and increasing precision.

Implementation

This section describes the three primary processes that underpin the solution: the ETL process, vector index creation, and the Retrieval-Augmented Generation (RAG) process.

1. ETL Process

The ETL (Extract, Transform, Load) process is the initial step where data from PDF documents containing Mexican tax laws, is extracted. The data is then transformed into a standardized format in the golden document, each golden document represents a chunk from the original PDF content enriched with bounding boxes of its position in the file, a short synthetic summary, and a set of synthetic questions related to the chunk.

ETL Process Architecture

ETL Data Flow

Documents JSON schema

2. Vector Index Creation

In the vector index creation step, the process creates the vectors and metadata to store in Pinecone. For each golden document, one vector is created for the synthetic summary, and another one for each synthetic question, the metadata is populated with the golden document's unique identifier for future retrieval and enrichment purposes.

3. RAG Process

The Retrieval-Augmented Generation (RAG) process utilizes the indexed vectors during the query phase. When a query is received, it is encoded into an embeddings vector to perform semantic search over the Pinecone database, with the retrieve data the chat process creates a contextualized prompt which is used to augment the chat completion response from the LLM.

Ok, ok, now show me the code!

erickbr15 / palaven-llm-sat

LLM RAG with openAI / Pinecone / Azure

PALAVEN - Project Documentation

1. Introduction

This project leverages .NET 6 (Long Term Support) to create a backend-focused solution incorporating OpenAI's Chat and Embeddings services, Pinecone's HTTP API, and Azure AI for PDF text extraction. This phase does not include web APIs or Azure Functions but features three console applications for data ingestion, vector index creation, and testing interactions with OpenAI's Chat API.

2. Requirements and Setup

System and Software Requirements

.NET 6 (LTS) environment.
Access to OpenAI APIs.
Pinecone HTTP API setup.
Azure AI services for text extraction.

3. System Architecture

4. OpenAI API Integration

Chat and Embeddings Services

The project integrates OpenAI's API through HTTP requests, utilizing the .NET IHttpClient for connectivity. Key components of this integration include:

Endpoints
- Chat completion endpoint: Utilized for 1. Extracting the article text, 2. Generating the synthetic questions about each ingested article, 3. Generating responses based on user inputs
- Embeddings creation…

View on GitHub

Helpful links

Finally...

I encourage you to consider this work as a foundational reference for your projects. The methodologies and technologies detailed here offer a robust framework for enhancing the capabilities of AI-driven systems, particularly in specialized domains.

Thanks for reading and happy coding!

Deep Learning in Agriculture: Leaf Classification Project Overview

Erick Badillo — Sun, 31 Mar 2024 13:23:21 +0000

Importance in Agriculture

In today’s agricultural sector, rapid and accurate identification of plant species and their health status is paramount. Diseases in crops can significantly impact yield and, consequently, the overall food supply. This project’s goal to classify leaf images and assess their health has far-reaching implications:

Early Disease Detection: By identifying diseased plants early, farmers can take prompt actions to mitigate the spread of disease, potentially saving large portions of crops.
Smart Farming: Integrating this technology with drones or automated systems can lead to efficient monitoring of large fields, enabling farmers to identify and address issues swiftly and on a much larger scale.

For a comprehensive understanding of this project, including details of the neural network architecture and code to train, evaluate, and make classifications, please visit my GitHub repository.

erickbr15 / dlpytorch-leaves-classification

The core of this project revolves around the development of specialized neural network models for two key tasks: species classification and health status assessment of plant leaves.

Custom ResNet18 Neural Network for Leaf Classification and Health Assessment

Project Methodology Overview

The core of this project revolves around the development of specialized neural network models for two key tasks: species classification and health status assessment of plant leaves.

Species Classification Model: A robust model was designed to identify the species of a plant from its leaf image. This model can classify leaves into various species, providing a fundamental layer of analysis.
Health Status Classification Models: For each species with sufficient data, an individual model was trained to determine the health status of the leaf. These models are capable of differentiating at least two classes, offering a deeper insight into the plant's condition.

Repository and Structure

The repository is structured as follows to facilitate navigation and understanding:

/models: Stores the trained model files, which can be used for direct predictions without the need for retraining.

Model File

Leaf

…

Model	File
Leaf

View on GitHub

Project Details and Neural Network Architecture

Approach

I developed a two-fold neural network model: the first to identify the plant species from leaf images, and the second to determine the health status of these identified species.
This dual-model approach enhances accuracy and provides a comprehensive understanding of plant health.

Deep Neural Network Architecture

Customized ResNet18 Model: The backbone of this project is a customized version of the well-known ResNet18 architecture.
Network Adaptation: The initial and final layers of ResNet18 were modified. The input layer was adjusted to accommodate the shape of the leaf image tensors. Similarly, the output layer was tailored to produce results in line with the number of classes in our dataset.
Application in Models: This adapted architecture forms the basis for both species identification and health status classification models.

Implementation details

Custom Dataset and Neural Network Architecture

Custom Dataset Creation: Developed a dataset that processes images into tensors of shape 3x224x224, conforming to the required input structure for neural network models.
Class Balance Analysis: Employed bar charts to assess the balance among classes, identifying those needing synthetic data augmentation.
Synthetic Data Generation: Crafted a data augmentation generator, allowing for customizable transformations to balance classes effectively.
DataLoader Setup: Established DataLoaders for training, validation, and testing, with data distribution set at 80% for training, 2% for validation, and 18% for testing.
Network Customization: Adapted ResNet18 architecture to suit our needs, modifying it to handle 3x224x224 tensors and configuring the output to manage 14 classes. Training Method and Process: Utilized an Adam optimizer with a 0.001 learning rate and Cross Entropy Loss function. The model underwent 10 epochs of training, focusing on experimental and validation purposes.
Infrastructure Note: Training was performed using CUDA/GPUs on Colab, with a recommendation to utilize the entire dataset on more advanced infrastructure for future iterations.

Performance and Predictive Analysis

Model Performance: Evaluated using the test dataset to ensure reliability and accuracy.
Predictive Methodology: Demonstrated through a coded example, showcasing how the persisted model can infer results from an image file.

Potential Applications

Imagine fields surveyed by drones, capturing images of crops. These images, when processed through our models, can provide valuable insights into the health of each plant. This real-time data can guide farmers to take immediate action wherever necessary, transforming the way we approach agriculture and crop management.

Why This Matters

This project is not just an academic exercise but a step towards revolutionizing agricultural practices using deep learning. By accurately identifying diseases and assessing plant health on a large scale, we can significantly increase efficiency in farming, leading to better crop yields and sustainable farming practices.

C# Gains Ground: Challenging Java's Dominance

Erick Badillo — Mon, 23 Oct 2023 20:43:19 +0000

In October 2023, C# is in close competition with Java in popularity, highlighting Microsoft's significant advancements with this language. What are the reasons behind this success, and what advantages do you see in continuing to use Java instead of C# for your projects? Would you be willing to objectively evaluate C# as a viable option for your projects in the future?

TIOBE Index

Here I leave you some very good things I see about Microsoft, .NET, and C#.

C# provides modern language features that allow programmers to write secure, readable, maintainable code with the required performance. What's new in C#

NET is a robust, cross-platform, free, and open-source ecosystem. .NET

Microsoft is highly committed to providing high-quality tools to create applications, so we have at hand Visual Studio Community Edition, which is free, and we have the Professional and Enterprise editions See the Visual Studio website.

We also have the free IDE Visual Studio Code that provides a good programming experience for developers who enjoy doing almost everything using commands and the console (bash, git bash, PowerShell, node, command prompt). See the Visual Studio Code website

These IDEs are extensible, so you can create your own extensions or use any of your preferences in the Marketplace

And, if you don't want to use these IDEs you can use a different one, or even you can create a .NET application using a notepad and compile it using the .NET API in a console. .NET CLI Overview

Finally, you can find extensive training for all levels, beginners, intermediates, and advanced profiles on the Microsoft Learn site. These trainings go from basic programming topics to architectural guidelines for enterprise applications.

Thank you for reading and happy coding!

Breaking Language Barriers with Microsoft Azure AI Translator

Erick Badillo — Wed, 04 Oct 2023 07:39:40 +0000

“Artificial intelligence is the new electricity.” - Andrew Ng

Hello!. Today, we embark on an exciting journey as we kickstart a series of short posts that will shed light on the remarkable capabilities of Azure AI.

This post serves as a demonstration of how you could abstract and implement Azure's AI services for referencing in your applications. Within this post, you will find a link to the code showcasing a lightweight version of Azure resources that you can utilize to deploy an AI-powered application in production environments. Hopefully, the provided designs and abstractions will prove useful as they are, and in any other case, I hope this work serves as a foundation for you to build upon in your own implementation.

What Not to Expect:
This post is not a step-by-step guide on how to use Azure's artificial intelligence services. It is also not a duplicate of Microsoft's official documentation, nor is it a copy of the code that can already be found in the examples provided by Microsoft for developers to implement their Azure AI services. If you wish to access Microsoft's resources, please take a look at the references section.

Let's have fun

In our interconnected world, effective communication transcends borders and languages. Microsoft Azure Translator, an AI-powered service, empowers developers to seamlessly bridge language gaps. Join me as we explore this powerful tool and its potential impact on your projects.

Azure Translator is designed to effortlessly translate text and spoken language between multiple languages, making it a vital tool for applications aiming to break down language barriers.

With Azure Translator, you can:
-Achieve highly accurate translations.
-Detect and identify languages in real-time.
-Translate entire documents while preserving formatting.
-Enable multilingual chatbots for superior customer support.
-Transliterate to different scripts

Show me the code!

You can find the demo code here

Explain me the design

The AI Azure services demo is a C# solution created in Visual Studio using .NET7.

The solution contains two assemblies:

Class library AzureAI.Poc.Services.Api
xUnit test project which contains:
- The app root composition code
- The integration test fixtures for each AI service (for this post, we are reviewing the Translator tests)

The solution uses the following Azure resources:

Azure Key Vault to store the AI Services API keys
Azure App Configuration (do I need to explain why we need this?). To store the application configurations.
Azure multi-AI service Please notice you can deploy the needed Azure resources using the ARM template that you can find in the GitHub repository

AzureAI.Poc.Services.Api

This assembly contains all the code you might use in your own implementations. The goal of this assembly is to show you and luckily bring you some ideas of how to add the AI Azure services to your applications. In this assembly you can find:

Common classes like the HTTP Proxy and the global configurations model.
The strong type model classes for the different AI services (where applicable)
The contract and implementation to connect to the Azure AI Rest services and get string JSON formatted responses. See the class TranslatorRestClient

public async Task<string> DetectAsync(IEnumerable<DetectRequest> 
request, CancellationToken cancellationToken)
{
    var route = _routeBuilder.BuildDetectRoute();
    var body = request.Select(r => new TranslatorRequestBody 
               { 
                   Text = r.Text 
               }).ToArray();

    var response = await PostAsync(route, body, 
                                   cancellationToken);        
    return response;
}

The contract and implementation to connect to Azure AI Rest services and get a strong type response. See the class TranslatorService

public async Task<DetectResult[]?> DetectAsync(
IEnumerable<DetectRequest> request, 
CancellationToken cancellationToken = default)
{
    var detectResult = await _translatorRestClient.DetectAsync(
                       request, cancellationToken);
    var result = JsonConvert.DeserializeObject<DetectResult[](
                 detectResult);

    return result;
}

Class view of the main classes involved in the translation functionality

Ok, ok, now show me how to invoke it!

Call the none strong-typed service

var request = new TranslateRequest
{
    FromLanguage = "en",
    ToLanguages = new List<string> {"es-MX"},
    IncludeAlignment = true,
    IncludeSentenceLength = true
};

request.Text.Add("Hello!, what's your name?");

var response = await _translatorRestClient.TranslateAsync(request,
 CancellationToken.None);

Call the strong-typed service

var request = new TranslateRequest
{
    FromLanguage = "en",
    ToLanguages = new List<string> { "es-MX" },
    IncludeAlignment = true,
    IncludeSentenceLength = true
};

request.Text.Add("Hello!, what's your name?");

var response = await _systemUnderTest.TranslateAsync(request, 
CancellationToken.None);

Final comments

In this Azure Translator Service demo, we have used the REST API to showcase the most used language translation capabilities. Stay tuned for more in these series, and start your own AI journey today.

References

Azure AI Translator Service:
[Online]. https://azure.microsoft.com/en-us/products/ai-services/ai-translator
Azure AI Translator APIs:
[Online]. https://learn.microsoft.com/en-us/azure/ai-services/translator/translator-text-apis?tabs=csharp
Azure AI Translator Documentation:
[Online]. https://learn.microsoft.com/en-us/azure/ai-services/translator/
More Training Resources:
[Online]. https://learn.microsoft.com

Multiple Linear Regression

Erick Badillo — Tue, 26 Sep 2023 06:23:46 +0000

Notebook code

Hello, this is my second post regarding Linear Regression. These posts aim to share the fundamentals of the programming and math concepts behind the basics of ML algorithms.

You can go post by post and follow the sequence, or you can read only what you are interested in. I hope you find interesting this amazing topic and get a good understanding of the basics behind ML algorithms.

I will visit Linear Regression, Classification, Clustering, and some concepts of feature engineering, regularization, overfitting, and underfitting. Therefore, I hope you enjoy reading and please let me know your thoughts.

In the previous post , I wrote about how Linear Regression is implemented from scratch using:

Cost function: Mean Squared Error (MSE)
Optimization function: Gradient descent
Python implementation from scratch using loop statements
One feature to predict the label

In this post, I use:

Cost function: Mean Squared Error (MSE)
Optimization function: Gradient descent
Python implementation from scratch using Numpy
Multiple features to predict the label
R-squared metric to measure our model performance

Let's get started

Linear regression assumes a linear relationship between the label and features. This means that we believe the relationship can be expressed as a straight-line equation:

y = f (x) = w x + b

Considering we have more than one feature and more than one test example, we can generalize the above equation as a prediction hypothesis to compute the vector $y^\hat{y}$ .

y^→=w→⋅X+b\overrightarrow{\hat{y}} = \overrightarrow{w}\cdot X+b

Another way to write the above equation would be:

[y1^⋮ym^]=[w1⋮wn]⋅[x11⋯x1n⋮⋱⋮xm1⋯xmn]+[⋮b⋮] \begin{bmatrix}\hat{y_1}\\\vdots\\\hat{y_m}\end{bmatrix} = \begin{bmatrix}w_1\\\vdots\\w_n\end{bmatrix}\cdot \begin{bmatrix}x_{11} & \cdots & x_{1n} \\\vdots & \ddots & \vdots \\x_{m1} & \cdots & x_{mn}\end{bmatrix} + \begin{bmatrix}\vdots\\b\\\vdots\end{bmatrix}

Now we have the hypothesis to predict the label from multiple features. Let's have fun writing some code and doing some math.

1. Data generation

We will use a pre-built library to create an ad-hoc dataset for our exercise. Using this library is very helpful because in this way we can skip the feature selection, and feature engineering processes.

Nevertheless, whenever you want to compute predictions with your own data take into consideration that you will need to run into the feature selection, and feature engineering processes to ensure your model will perform well with unknown data.

Libraries import

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
from sklearn.metrics import r2_score

Data generation

X, y = make_regression(n_samples = 200, n_features = 3,
random_state = 42, noise = 50)
y = y.reshape(200, 1)

2. Data visualization

Features histogram

sns.histplot(X[:, 0], kde=True, label='Feature 1')
sns.histplot(X[:, 1], kde=True, label='Feature 2')
sns.histplot(X[:, 2], kde=True, label='Feature 3')
plt.xlabel('Feature Values')
plt.ylabel('Frequency')
plt.legend()
plt.show()

3. Training/Test sets creation

X_train, X_test, y_train, y_test = train_test_split(X, y,
test_size=0.2, random_state=42)

4. Model implementation

Cost function

\begin{equation}J(w,b) = \frac{1}{2m}\sum_{i=1}^{m}(f_{w,b}(x^{(i)})-y_{(i)})^{2}\end{equation}

def compute_loss(X, y, w, b, m):
    y_hat = np.dot(X, w) + b
    loss = (1 / (2* m)) * np.sum((y_hat - y)**2)
    return loss

The compute gradients function

\frac{\delta J(w,b)}{\delta w} = \frac{1}{m}\sum_{i=1}^{m}(f_{w,b}(x^{(i)})-y^{(i)})x^{(i)}

\frac{\delta J(w,b)}{\delta b} = \frac{1}{m}\sum_{i=1}^{m}(f_{w,b}(x^{(i)})-y^{(i)})

def compute_gradients(X, y, w, b, m):
    y_hat = np.dot(X, w) + b
    error = y_hat - y
    dj_dw = (1 / m) * np.sum(error * X, axis = 0)
    dj_db = (1 / m) * np.sum(error)        
    return dj_dw, dj_db

The optimization function (gradient descent)

w-\alpha\frac{\delta J(w,b)}{\delta w}

b-\alpha\frac{\delta J(w,b)}{\delta b}

def compute_optimization(learning_rate, w, b, dj_dw, dj_db):
    w_new = w - (learning_rate * dj_dw)
    b_new = b - (learning_rate * dj_db)
    return w_new, b_new

The fit(training) function

def fit(epochs, learning_rate, X, y, w, b, debug):                
    m, n = X.shape

    losses = list() #tracking purposes

    for epoch in range(epochs):

        loss = compute_loss(X, y, w, b, m)        
        losses.append(loss)        

        dj_dw, dj_db = compute_gradients(X, y, w, b, m)            
        w_new, b_new = compute_optimization(learning_rate, 
w, b, dj_dw, dj_db)

        w = w_new
        b = b_new

        # print
        if (debug and (epoch % 50) == 0):
            print(f"epoch: {epoch}, loss: {loss}")            

    return w, b, losses

Training the model

epochs = 500
learning_rate = 0.01

m, n = X_train.shape

w = np.random.rand(n,1)
b = np.random.rand(1)

w, b, loss = fit(epochs, learning_rate, X_train, y_train, w, b, True)
w, b

5. Prediction and model evaluation

Prediction function

def predict(X):
    return np.dot(X, w[0]) + b

Prediction and model evaluation using R-squared

R-squared ( $R^2$ is a metric that ranges from 0 to 1 and is sometimes expressed as a percentage between 0% and 100%. It represents the proportion of variance in the dependent variable (target) that is explained by the regression model. In other words, it indicates how much variability in target values is captured by the model.

R-squared interpretation

$R^2 = 0,$ The model explains no variability in the data and is essentially useless for making predictions.

$R^2 = 1,$ The model explains all the variability in the data and makes perfect predictions. This is rare in practice and could be a sign of overfitting.

$0 < R^2 < 1,$ The better the model fits the data. A higher value indicates that the model can explain a greater proportion of the variability in the target values.

y_pred = predict(X_test)
score = r2_score(y_test, y_pred)
score
0.7072394118821352

Final comments

In this journey through multiple linear regression, we embarked on a hands-on exploration of the fundamental concepts, implementation from scratch using Python and NumPy, and performance evaluation using the Mean Squared Error (MSE) metric and the R-squared ( $R^2$ ) score. Achieving an R-squared value of 0.707 demonstrates that our model captures a substantial portion of the data's variability, indicating a meaningful relationship between our predictor variables and the target. This journey not only allowed us to gain insights into the inner workings of regression modeling but also showcased the power of building predictive models from the ground up. Armed with these skills, we're well-equipped to tackle more complex regression problems and fine-tune our models for even better performance.

References

[1] A. Géron, "Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow," O'Reilly Media, Inc., Oct. 2022.

[2] I. Goodfellow, Y. Bengio, and A. Courville, "Deep Learning," The MIT Press.

Simple Linear Regression (basic overview)

Erick Badillo — Sun, 03 Sep 2023 07:41:45 +0000

This exercise shows the basics of the Linear Regression algorithm using:

MSE and Gradient Descent
One feature
Code written in Python

_You can download the notebook from https://github.com/erickbr15/ml-notebooks/blob/main/ml-linear-regression-simple-one-feature.ipynb

Linear regression is a fundamental statistical technique used in data analysis and machine learning for modeling the relationship between a dependent variable (label) and one or more independent variables (features). It is a versatile tool with applications in various fields, from economics to biology and beyond.

How Linear Regression Works:

Data Collection: Start by collecting a dataset with both labels and features. This process is also known as data-generating process.
Model Building: Choose the appropriate type of linear regression (simple or multiple) based on the problem at hand. Calculate the best-fitting line using techniques like the least squares method.
Model Evaluation: Assess the goodness of fit by examining metrics such as the coefficient of determination (R-squared), which tells you how well the model explains the variance in the data.
Prediction: Once you have a reliable model, you can use it to make predictions. For instance, you can predict house prices for new properties based on their sizes using the established linear relationship.

1. Data-generating process

The training and test data are generated by a probability distribution over datasets called the data-generating process. We tipically make a set of assumptions known collectively as the i.i.d. assumptions. These assumptions are that the examples in each dataset are independent from each other, and that the traning set and test set are identically distributed drawn from the same probability distribution as each other.

For this excercise we will use the library sklearn.datasets to generate the training and test sets.

#Import necessary libraries
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt
import numpy as np

#Data-generating
X, y = make_regression(n_samples = 150, 
n_features = 1, random_state = 42, noise = 50)

print(f" X.shape = {X.shape}")
print(f" y.shape = {y.shape}")

 X.shape = (150, 1)
 y.shape = (150,)

# EDA - Exploratory Data Analysis
plt.scatter(X, y)
plt.show()

Split dataset into training and test sets

X_train, X_test, y_train, y_test = train_test_split(X, y,
 test_size = .10, random_state = 42)

print(f" X_train.shape = {X_train.shape}")
print(f" y_train.shape = {y_train.shape}")

 X_train.shape = (135, 1)
 y_train.shape = (135,)

plt.scatter(X_train, y_train, label="Training set")
plt.scatter(X_test, y_test, label="Test set")
plt.legend()

Model building

Linear regression assumes a linear relationship between the label and feature variables. This means that we believe the relationship can be expressed as a straight line equation:

f (x) = w x + b

Initialize a random w and a random b. This task will allow us to define a starting point to optimize the weigths and bias and find the model that better fits the training data set.

m, n = X_train.shape
y_train = y_train.reshape(m, 1)
w = np.random.rand(1)
b = np.random.rand(1)

print(f"X_train shape={X_train.shape}")
print(f"y_train shape={y_train.shape}")
print(f"m={m}")
print(f"w={w}")
print(f"b={b}")

X_train shape=(135, 1)
y_train shape=(135, 1)
m=135
w=[0.89389793]
b=[0.45829251]

plt.scatter(X_train, y_train, label = "Training dataset")
plt.plot(X_train, 
np.dot(X_train, w) + b, color="red", label="Starting model")
plt.legend()
plt.show()

Now that it is defined how the label will be predicted from the feature using the model $y^=wx+b\hat{y} = wx + b$ we need a definition of our performance measure. One way of measuring the performance of the model is to compute the mean squared error of the model on the test set. If $y^\hat{y}$ gives the predictions of the model on the test set, then the mean squared error is given by

MSEtest=J(w,b)=12m∑im(fw,b(x(i))−y(i))2MSE_{test}=J(w,b)=\frac{1}{2m}\sum_{i}^{m}(f_{w,b}(x^{(i)})-y^{(i)})^{2}

This measure is called the cost J(w,b) and intuitively, one can see that this error measure decreases to 0 when $y^=y\hat{y} = y$ . We can also see that the error increases whenever the Euclidean distance between the predictions and the targets increases.

We need to improve the weights w in a way that reduces $MSE_{test}$ . One intuitive way of doing this is just to minimize the mean squared error on the training set. To minimize the cost function and find the global minimum we will use the gradient descent.

The gradient descent algorithm is described as repeat until convergence:

w=w−αδJ(w,b)δww=w-\alpha\frac{\delta J(w,b)}{\delta w}

b=b−αδJ(w,b)δbb=b-\alpha\frac{\delta J(w,b)}{\delta b}

where parameters $w, b$ are updated simultaneously and $α\alpha$ is the learning rate.

Until now we have:

Linear regression model: $f_{w,b}(x) = wx + b$
Cost function: $\frac{1}{2m}\sum_{i=1}^{m}(f_{w,b}(x^{(i)})-y_{(i)})^{2}$
Gradient descent algorithm

w=w−αδJ(w,b)δww=w-\alpha\frac{\delta J(w,b)}{\delta w}

b=b−αδJ(w,b)δbb=b-\alpha\frac{\delta J(w,b)}{\delta b}

Now, let's do some math to get $δJ(w,b)δw\frac{\delta J(w,b)}{\delta w}$

$\frac{\delta}{\delta w}\frac{1}{2m}\sum_{i=1}^{m}(f_{w,b}(x^{(i)})-y^{(i)})^{2}$

$\frac{\delta}{\delta w}\frac{1}{2m}\sum_{i=1}^{m}(wx^{(i)}+b-y^{(i)})^{2}$

$\frac{1}{2m}\sum_{i=1}^{m}(wx^{(i)}+b-y^{(i)})2x^{(i)}$

$\frac{1}{m}\sum_{i=1}^{m}(f_{w,b}(x^{(i)})-y^{(i)})x^{(i)}$

Now, to get $δJ(w,b)δb\frac{\delta J(w,b)}{\delta b}$

$\frac{\delta}{\delta b}\frac{1}{2m}\sum_{i=1}^{m}(f_{w,b}(x^{(i)})-y^{(i)})^{2}$

$\frac{\delta}{\delta b}\frac{1}{2m}\sum_{i=1}^{m}(wx^{(i)}+b-y^{(i)})^{2}$

$\frac{1}{2m}\sum_{i=1}^{m}(wx^{(i)}+b-y^{(i)})2$

$\frac{1}{m}\sum_{i=1}^{m}(f_{w,b}(x^{(i)})-y^{(i)})$

Code implementation

To implement the Linear Regression using gradient descent and MSE, we will need the next functions:

function to compute the cost
function to compute the gradient
function to compute the gradient descent (fit)

def compute_cost(X, y, w, b, m):
    cost_sum = 0 
    for i in range(m): 
        f_wb = w * X[i] + b   
        cost = (f_wb - y[i]) ** 2  
        cost_sum += cost  
    total_cost = (1 / (2 * m)) * cost_sum

    return total_cost

def compute_gradient(X, y, w, b, m):
    dj_dw = 0
    dj_db = 0

    for i in range(m):  
        f_wb = w * X[i] + b 
        dj_dw_i = (f_wb - y[i]) * X[i] 
        dj_db_i = f_wb - y[i] 
        dj_db += dj_db_i
        dj_dw += dj_dw_i 
    dj_dw = dj_dw / m 
    dj_db = dj_db / m 

    return dj_dw, dj_db

def fit(X, y, w, b, m, learning_rate, epochs):
    W = np.array([])
    B = np.array([])
    L = np.array([])

    working_w = w
    working_b = b

    for epoch in range(epochs):
        dj_dw, dj_db = compute_gradient(X,y,working_w, working_b, m)

        working_w = working_w - learning_rate * dj_dw
        working_b = working_b - learning_rate * dj_db

        loss = compute_cost(X, y, working_w, working_b, m)

        L = np.append(L, loss)
        W = np.append(W, working_w)
        B = np.append(B, working_b)

    return working_w, working_b, L, W, B

optimized_w, optimized_b, cost_h, w_h, b_h = 
fit(X_train, y_train, w, b, m, 0.1, 100)

print(optimized_w)
print(optimized_b)

[97.20911222]
[8.58244015]

Plotting the cost function and the progress history of the gradient descent

plt.scatter(np.linspace(1, 100, 100), cost_h)
plt.title("Cost minimization over iterations")
plt.xlabel("Iterations")
plt.ylabel("J(w,b)")
plt.show()

fig, axes = plt.subplots(1, 2, figsize=(15, 5))

axes[0].plot(w_h, cost_h)
axes[0].set_title('w , J(w,b)')
axes[0].set_xlabel('w')
axes[0].set_ylabel('J(w,b)')

axes[1].plot(b_h, cost_h)
axes[1].set_title('b, J(w,b)')
axes[1].set_xlabel('b')
axes[1].set_ylabel('J(w,b)')

Definition of prediction function using the model

def predict(X):
  return np.dot(X, 97.20911222) + 8.58244015

Test the model

plt.scatter(X_train, y_train, color="green", label="Train")
plt.scatter(X_test, y_test, color="red", label="Test")
plt.plot(X_test, predict(X_test), color="blue", label="Predictions")

plt.title("Train, Test and Predictions graph")
plt.xlabel("X")
plt.ylabel("Y")

plt.legend(loc='upper left')

plt.show

fig, axes = plt.subplots(1, 2, figsize=(15, 5))

axes[0].scatter(X, y)
axes[0].plot(X, np.dot(X, w) + b, color="red")
axes[0].set_title('Starting model (not optimized)')
axes[0].set_xlabel('X (feature)')
axes[0].set_ylabel('Y (label)')

axes[1].scatter(X,y)
axes[1].plot(X, predict(X), c = "r")
axes[1].set_title('Linear regression model (optimized)')
axes[1].set_xlabel('X (feature)')
axes[1].set_ylabel('Y (label)')

plt.show()

Linear regression model ( $\to f(x)\to prediction$ )

$f (x) = w x + b$

$y^=wx+b\hat{y}=wx+b$

$y^=97.20911222x+8.58244015\hat{y}=97.20911222x+8.58244015$

Final comments

While linear regression is a powerful tool, it has limitations. It assumes a linear relationship, which may not hold for all problems. Additionally, it can be sensitive to outliers and may not perform well when the data has a complex, nonlinear structure.

In conclusion, linear regression is a foundational technique in data analysis and machine learning that helps us model relationships between variables and make predictions. It's a valuable tool in a data scientist's toolkit and serves as a basis for more advanced regression techniques.

Linear regression has a wide range of applications:

Finance: Predicting stock prices based on historical data.
Marketing: Analyzing the impact of advertising expenditure on sales.
Medicine: Predicting patient outcomes based on clinical variables.
Environmental Science: Studying the relationship between pollution levels and health effects.

In future posts I will come with:

the metrics to evaluate the model,
a code implementation to use optimized algorithms to operate over vectors and matrix

Thank you for reading and happy coding!

References

[1] A. Géron, "Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow," O'Reilly Media, Inc., Oct. 2022.

[2] I. Goodfellow, Y. Bengio, and A. Courville, "Deep Learning," The MIT Press.

Transformative Applications of AI in Software Engineering

Erick Badillo — Sat, 19 Aug 2023 11:17:14 +0000

Artificial Intelligence (AI) has emerged as a transformative tool in the realm of Software Engineering (SE), revolutionizing how software development activities are approached. This overview delves into the manifold ways in which AI techniques are applied across various SE domains, ushering in automation, efficiency, and innovation.

1. Software Requirements: Efficient Classification

In large-scale projects, requirements can be complex and voluminous, making manual management challenging. This is where AI comes into play. Algorithms like Naïve Bayes, Support Vector Machine (SVM), and Deep Learning have proven effective in automatically extracting and classifying requirements. This improves accuracy and saves time for software engineers.

Some tools:

IBM Engineering Requirements Quality Assistant

2. Software Design: Architecture Recovery

During the early stages of development, design artifacts are crucial to guide the process. However, these artifacts can be incomplete or outdated. This is where ML shines by offering solutions for design challenges. Algorithms like k-Nearest Neighbors, Random Forest, and SVM help recover architecture and address the challenges posed by continuously evolving software.

Some tools:

CodeGraph - Microsoft Visual Studio IntelliCode
CodeMap - Microsoft Visual Studio Enterprise
JArchitect

3. Software Construction: Intelligent Component Reuse

In the realm of software construction, the practice of intelligently reusing software components to expedite development has gained significant traction. However, the challenge lies in accurately predicting which components should be reused, considering the intricate interplay of factors involved. This is where the power of ML and even Generative AI like GPT-3, 3.5, 4 comes into play.

ML algorithms, such as Logistic Regression, Naïve Bayes, and Decision Tree, are instrumental in predicting the optimal reuse of software components. By analyzing historical data and understanding the context of the project, these algorithms can forecast the most pertinent components for reuse. However, the landscape of AI-powered software construction is evolving rapidly, with Generative AI stepping into the spotlight.

Generative AI leverages natural language processing (NLP) to generate human-like text. What's fascinating is that it's now capable of producing code snippets directly from NLP-based descriptions. This innovation further enriches software construction by allowing developers to translate high-level requirements and concepts into tangible code components.

The marriage of traditional ML algorithms and cutting-edge Generative AI demonstrates the capacity to revolutionize software construction. With intelligent component reuse and code generation becoming a reality, software engineers have an array of tools at their disposal to expedite development, optimize resource utilization, and drive innovation forward.

This synthesis of established ML techniques and emerging Generative AI capabilities underscores the ever-expanding horizons of software engineering, showcasing how technology's collaborative efforts reshape development practices for the better.

Some tools:

SonarQube
GitHub Copilot
openAI chatGPT

4. Software Testing: Defect Prediction

ML is proving its worth in software testing as well. ML-based approaches focus on defect and bug prediction, as well as functional testing automation. By analyzing code and process metrics, development teams can identify error-prone areas and focus their efforts accordingly. Both traditional and Deep Learning algorithms like k-NN, kMeans, Naïve Bayes, and Neural Networks are successful in addressing software testing challenges.

Some tools:

DeepCode AI

5. Software Maintenance: Refactoring and Fault Prediction

Refactoring and fault prediction are key areas where ML techniques are applied. Refactoring helps prioritize changes and improve existing code. Fault prediction identifies components that may need modifications to fix defects. Algorithms like Logistic Regression, Naïve Bayes, Multilayer Perceptron, and Artificial Neural Networks show promise in enhancing software maintenance.

Some tools:

Microsoft IntelliCode
ReSharper by JetBrains
VSSonarQube

Conclusion: Towards a Better Future for Software Engineering

The AI is significantly transforming Software Engineering. Through automation and improvement in various areas, software professionals can achieve higher-quality and more efficient results. As AI techniques continue to evolve, it's exciting to consider how they will further drive innovation in Software Engineering and the entire tech industry.
In summary, AI is proving to be a powerful tool in the world of Software Engineering, enhancing tasks from requirement definition to software quality. Stay tuned for emerging trends, as the future of Software Engineering is being shaped by artificial intelligence.

References

[1] O. Borges, M. Lima, J. Couto, B. Gadelha, T. Conte, and R. Prikladnicki, "ML@SE: What do we know about how Machine Learning impact Software Engineering practice?" 2022 17th Iberian Conference on Information Systems and Technologies (CISTI)