Intro
Recently I completed a project from data science boot camp and learned how the transformation of numerical information is helpful in finding a regression model. On this page, I like to focus on how the transformation of numerical data can give better validation to the model instead of focusing on how the model looks and can be used to predict a value.
I will go over the data source and data structure briefly and give a quick explanation of what methods are used in this post. I will also include some python code and mathematical formulas to aid understanding.
Data
I am using data from the project I had completed, but the data is from a real-life and contains information on house sales data in King County, Washington. The information includes house prices and multiple house features. Below is the list of variables used in the analysis.
Dependent Variable
House Price
Independent Variables
Numerical
Living space in squared-feet
Lot size in squared-feet
Year built
The number of floors*
* A separate explanation of why I defined it as numerical is at the end of the post.
Categorical
Binaries
Waterfront
View presence
Renovation condition
Basement presence
Multi-categorical
Maintenance condition
House grade
Methods
This section is just to give you an idea of what I have done for the results. If these look familiar to you or don't interest you, then you can just skip to the result section. Checking the results before reading this section might give you an idea more easily of why I am posting this.
Assumptions
There are several ways to validate the model, and I like to go over the four major assumptions to validate the model. The assumptions are linearity, normality, homoscedasticity, and multicollinearity. I chose this method because it can be explained visually, and visualization is more helpful in explaining the concept than just lots of words and numbers.
1. Linearity
It is important to check the linearity assumption in the linear regression analysis. As polynomial transformation has not been applied in this analysis, the expected house price (dependent variable) will be compared to the raw value of the house price.
Below is the python code I used. The purpose of sharing the code is to give some idea of how the graph is created.
# split whole data to training and test data
from sklearn.model_selection import train_test_split
X = independent_variables
y = house_price_column
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)
# find the model fit
from sklearn.linear_model import LinearRegression
model = LinearRegression()
model.fit(X_train, y_train)
# Calculate predicted price using test data
y_pred = model.predict(X_test)
# Graphing part
import matplotlib.pyplot as plt
fig, ax = plt.subplots()
perfect_line = np.arange(y_test.min(), y_test.max())
ax.plot(perfect_line, linestyle="--", color="orange", label="Perfect Fit")
ax.scatter(y_test, y_pred, alpha=0.5)
ax.set_xlabel("Actual Price")
ax.set_ylabel("Predicted Price")
ax.legend();
2. Normality
The normality assumption is related to the normality of model residuals. This is checked using a QQ plot.
import scipy.stats as stats
residuals = y_test - y_pred
sm.graphics.qqplot(residuals, dist=stats.norm, line='45', fit=True);
3. Homoscedasticity
The assumption of homoscedasticity checks the dependent variables against the dependent variables and sees if values are dispersed without any pattern. This assumption is also related to the residuals.
fig, ax = plt.subplots()
residuals = y_test - y_pred
ax.scatter(y_pred, residuals, alpha=0.5)
ax.plot(y_pred, [0 for i in range(len(X_test))])
ax.set_xlabel("Predicted Value")
ax.set_ylabel("Actual - Predicted Value");
4. Multicollinearity
The assumption of multicollinearity checks dependency between independent variables. It is best to have independent variables independent of one another as much as possible.
from statsmodels.stats.outliers_influence import variance_inflation_factor
vif = [variance_inflation_factor(X_train.values, i) for i in range(X_train.shape[1])]
pd.Series(vif, index=X_train.columns, name="Variance Inflation Factor")
Transformations of numerical variables
Log transformation
This part is simple. All values in the numerical columns are natural-logged.
Normalization
The below formula shows a value in a numerical variable is subtracted by the mean of the variable, and then the subtracted value is divided by the standard deviation of the variable.
Results
Here is the fun part. You can just relax and see how graphs and scores change.
The Raw Data - no transformation
1. Linearity
I see several outliers. Some linearity is observed only on the left side.
2. Normality
Only 1/3 of the dots are on the red line.
3. Homoscedasticity
A clear pattern is observed.
4. Multicollinearity
Only scores below 5 are accepted. About half of the scores are not acceptable.
sqft_living 8483.406359
sqft_lot 1.200729
floors 14.106084
waterfront 1.085728
view 1.344477
yr_built 72.300842
is_renovated 1.157421
has_basement 2.175980
condition_Fair 1.038721
condition_Good 1.668386
condition_Very Good 1.295097
grade_11 Excellent 1.530655
grade_6 Low Average 5.129509
grade_7 Average 14.142031
grade_8 Good 8.261598
grade_9 Better 3.446987
interaction 8460.117213
Log Transformation
1. Linearity
It shows much better linearity. The linearity of the dots have a slightly lower slope than the perfect line.
2. Normality
There is a small kurtosis, but the majority of the dots are on the red line.
3. Homoscedasticity
This looks better, too. A slight pattern is observed on the right side.
4. Multicollinearity
Several scores are still too high.
sqft_living 471370.972327
sqft_lot 155.772190
floors 4.052275
yr_built 922.928871
waterfront 1.086052
view 1.337069
is_renovated 1.146855
has_basement 2.438983
condition_Fair 1.042784
condition_Good 1.668688
condition_Very Good 1.283740
grade_11 Excellent 1.468962
grade_6 Low Average 5.221791
grade_7 Average 12.895007
grade_8 Good 7.519577
grade_9 Better 3.355969
interaction 469074.416388
Log Transformation and Normalization
1. Linearity
The slope is slightly better and closer to the perfect line.
2. Normality
I don't see much difference from the previous graph.
3. Homoscedasticity
I don't see much difference from the previous graph.
4. Multicollinearity
All of the scores are now acceptable. This is a huge difference!
sqft_living 3.001670
sqft_lot 1.552016
floors 2.046914
yr_built 1.758294
waterfront 1.086293
view 1.313341
is_renovated 1.148279
has_basement 2.441147
condition_Fair 1.042169
condition_Good 1.647135
condition_Very Good 1.281906
grade_11 Excellent 1.278034
grade_6 Low Average 1.939542
grade_7 Average 2.077564
grade_8 Good 1.609822
grade_9 Better 1.440610
interaction 1.374175
Conclusion
Transformations helped to keep (i.e. not reject) the four assumptions. Visualizations seem clear enough to guide you to study what was improving.
Extra
The number of floors
This is kind of out of the major topic in this post, but this decision can be crucial to the overall regression analysis. Let me begin with the value counts of the floor information.
1.0 10673
2.0 8235
1.5 1910
3.0 611
2.5 161
3.5 7
The left column shows the data has a range of the floor counts from 1 through 3.5. The model might make more sense to treat this variable as a categorical variable. However, what if one likes to predict a house price that has 4 floors? This question or problem can be solved if this information is treated as numerical. I think this is a matter of the goal of the analysis.
Top comments (0)