Understanding Logistic Regression

Hello dear reader and welcome to another article on my take in different data science elements. This article will focus on logistic regression and will end with a look at sckitlearn's logistic regression model.

What is Logistic regression?

Logistic regression is a statistical method used to model the probability of a binary outcome based on one or more predictor variables. Despite its name, it's a classification algorithm rather than a regression one because it predicts the probability of the binary outcome instead of directly predicting the outcome itself.

Basically imagine you had to make a program that could tell apart an apple based on its color with the intention to have it instantly classify an apple just like you do when you see a fruit and say, "That's an apple!" or "That's not an apple!"

Lets breakdown how you would accomplish that.The way it would learn would be from interaction and learning from examples.You would feed it(no pun intended or is it😉) many apples and many non-apples (like oranges and bananas) and tell it their colors and sizes.

You would then have it make random guesses.But you want it to get better, so you tell it when it's wrong and how wrong it is.
Every time the program makes a wrong guess, it learns from its mistake and tries to adjust its guess to be closer to the correct answer.

The program would require a special formula that would help it make better guesses over time. This formula would look at the colors and sizes of the fruits it has interacted with and decide the probability of something being an apple or not.

Now imagine drawing a line on a piece of paper. On one side of the line, everything is considered an apple, and on the other side, everything is not an apple and asking the program to draw this line as accurately as possible based on the colors and sizes.

The more examples you feed the program, the better it gets at drawing the line. Eventually, it gets so good that it can look at a new fruit it hasn't seen before and make a very good guess if it's an apple or not.

That in a nutshell is logistic regression.It is having a model that learns from its mistakes and gets better with practice, just like how you learn to guess things better the more you see them.

Model Representation
Let's dive into the technicals of the model. Logistic regression models the relationship between the predictor variables and the binary outcome using the logistic function (sigmoid function). The logistic function ensures that the predicted probabilities lie between 0 and 1, which is essential for binary classification.

The decision boundary is the line that separates the two classes (0 and 1) in the feature space. It's determined by the weights (coefficients)0 learned during the training phase. is h0(x) is greater than or equal to 0.5, the model predicts class 1; otherwise, it predicts class 0.

During training, the parameters θ are learned by minimizing a cost function, typically the cross-entropy loss function. Gradient descent or other optimization algorithms are used to minimize this cost function. The optimization process adjusts the parameters to maximize the likelihood of the observed data given the model.

Advantages of Logistic Regression:

Interpretability: Logistic regression provides interpretable results. The coefficients associated with each predictor variable indicate the impact of that variable on the predicted probability of the outcome.

Efficiency: Logistic regression is computationally efficient, making it suitable for large datasets with many features. It can handle high-dimensional data with relative ease.

Robustness to Noise: Logistic regression can perform well even in the presence of irrelevant features or noisy data. It's less prone to overfitting compared to more complex models.

Applications of Logistic Regression:

Medical Diagnosis: Logistic regression is widely used in medical research for predicting the likelihood of disease based on patient characteristics, such as symptoms, demographic information, and medical history.You can look at my approach using the Breast Cancer Wisconsin (Diagnostic) Data Set to predict wheter a tumor is malignant or benign using logistic regression here.

Credit Scoring: Banks and financial institutions use logistic regression to assess the creditworthiness of loan applicants. It helps in predicting the probability of default based on factors such as income, credit score, and debt-to-income ratio.

Marketing Analytics: Logistic regression is used in marketing analytics for predicting customer behavior, such as whether a customer will respond to a marketing campaign, make a purchase, or churn.

Risk Management: Logistic regression is employed in various risk assessment tasks, such as predicting the likelihood of insurance claim fraud, identifying high-risk individuals for preventive interventions, and assessing the probability of accidents or failures in engineering systems.

If you've made it here then it's time for the good stuff. It's time to introduce logistic regression using sckitlearn.

As is the norm we start by importing the libraries. (This example does not include any analysis using pandas) .For this example, let's consider the classic Iris dataset, which contains features of iris flowers and their corresponding species.

# Importing necessary libraries
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_iris

We then load the iris dataset. This can also be achievedusing pandas.

iris = load_iris()
X = iris.data
y = iris.target

Then we split the dataset.

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

Let us now initialise a Logistic Regression model

model = LogisticRegression()

Let us now train the model on the training data and make predictions on the testing data

model.fit(X_train, y_train)
predictions = model.predict(X_test)

Now we can evaluate the model's performance

accuracy = model.score(X_test, y_test)
print("Accuracy:", accuracy)

It is that simple really.What in my opinion makes logistic regression stand out is that it is powerful and versatile with applications across various domains yet it is simple to work with and to interpret . While it's well-suited for binary classification tasks, it can also be extended to handle multi-class classification with techniques like one-vs-rest or softmax regression. Understanding logistic regression provides a solid foundation for more advanced machine learning methods and helps practitioners make informed decisions in real-world scenarios.

Thank you for the read. Happy coding 😊😊