OLS Linear Regression Basics with Python’s Scikit-learn (2023)

One of the oldest and most basic forms of predictions, linear regressions are still widely used in many different fields to extrapolate and interpolate data.

In this article, I’ll explain the basics of how and when to use them, with the help of Python’s Scikit-learn.

Linear regression was firstly published by the french mathematician Legendre in 1805. With its first applications in astronomy.

This modelling technique tries to predict the value of a variable dependent on another variable.

For example, a crop in which the yield depends on the amount of rain or a product in which the sales depend on their price.

As a side note — maybe you were fooled by the joyful caricature of our friend here, but let me tell you, this is the guy who figured out how to hit stuff with cannonballs. He did so by using air resistance, the projectile’s initial velocity, and the angle of the cannon to calculate the trajectory of the shot.

For this example, I’ll be using Jupyter Notebooks to run my code, Pandas for data wrangling, Numpy for math operations, Matplotlib for the visualizations, and Scikit learn for statistics.

Let’s start with small steps and build a straightforward dataset to test our linear model.

Our data is symmetrical, x = y, so it should be easy for our regression to predict that the value of y for x(10) is 10.

It’s important to remember that the “prediction” is the application of our conclusion in the broader population, it’s an extrapolation of the results. The conclusion itself, or the product of regressions, is a formula describing the relationship between the variables.

X = coefficient * y

let’s try it!

First, we define our model, with linear_model.LinearRegression(), then we fit our model with .fit(x, y).

Now, let’s check our coefficient.

It’s 1. This means our formula is: x = 1 * y

Let’s predict the next values.

It works!

*Scikit requires X to be a matrix (2d array), this is because you can have more than one explanatory variable in your model.

Ok, besides having a perfectly linear relationship, our data starts at the same point, but what happens when it doesn’t?

That’s not because linear regressions can’t handle it, it’s because we’re missing one crucial part in our formula. The intercept!

The intercept is a constant value that adds to our formula regardless of the values of the variables. So let’s remove that “fit_intercept=False” and try again.

y = intercept + coefficient * x
y = (-10) + (1* x)

Awesome, we know how to apply a basic linear regression, and we know how to interpret its results.

But, what is the model doing?

So, first things first, the type of regression we’re using is OLS — Ordinary Least Squares.

Let’s see how Scikit describes this model.

LinearRegression fits a linear model with coefficients w = (w1, …, wp) to minimize the residual sum of squares between the observed targets in the dataset, and the targets predicted by the linear approximation

Alright, pretty condensed statement over there. Let’s try to distil what it is trying to say.

*The way our model evaluates the results is often referred to as its metrics.

This model looks for the residual sum of squares. The residual is simply the difference between the predicted value by the model and the actual value.

Take a look at this example:

So the linear regression represented by the blue line indicates the prediction, but we can see that the data points are not exactly in the blue line. The difference between those points is the so-called residual.

Ok, so the residual sum of squares is the sum of the differences between our prediction and our actual data, it’s squared so we can avoid negative numbers in our sum — So an OLS linear regression tries to minimize those differences.

Metrics are very significant when we’re comparing different models.

If we’re looking at different models, trying to predict the same data we can say that the model with the lowest residual sum of squares is the most accurate.

It’s convenient to get those metrics with Scikit. We just need the metrics class.

The most common metrics for linear regressions are the R2 score and the mean squared error, but I highly suggest that you check the other available metrics in the documentation and think about the best metric for your purpose.

Maybe you don’t mind having a higher average residual, but you can’t afford to have extremely high errors, so a max_error(y_true, y_pred) could be more suitable for your needs.

We don’t have residuals in this regression, so it’s Mean Squared Error will be zero.

R2 is the coefficient of determination, where a perfect score would be 1, decreasing as the predicted points move farther from the actual values.

*Even though we’re talking about squares, this is a coefficient, so it can be negative.

(Video) Simple Linear Regression in Python - sklearn

Cool, we have seen how to build a simple OLS model, how to fit our data to it, extrapolate on our conclusions, and the metrics we can use to compare the models.

Not quite yet, Neo.

Now, let’s get to one of the most important and most forgotten parts of this methodology. The ASSUMPTIONS!

Assumptions are like pre-requisites; we have to check to know if this method will work in our data.

Breaking assumptions, or running your model regardless of them, may result in consequences. Some may have higher impacts than others in your prediction, and I suggest some further research about those assumptions if you’ll be heavily relying on this model, just as I would recommend for any statistical model.

For the sake of simplicity, let’s take a look at the most meaningful of them.

Linearity — This means the relationship between the two variables must be linear.

Independence — The observations should be independent of each other. The amount of daily rain doesn’t depend on how much it rained in the next or the previous day, but the amount of water in a reservatory is highly dependent on its prior values.

Homoscedasticity — This means our residuals should have a consistent variance. Time-series usually have problems with this. Initially, we have consistent predictions, but the error gets higher as we move further in time.

Normality — The errors should be normally distributed. Here, we’re mostly looking for significant deviations from a normal distribution, like a very right or left-skewed error distribution.

Sample Size — The sample should have at least 20 observations. We’ll break this assumption here for simplification.

That’s a lot, now let’s try an OLS model with multiple explanatory variables!

It’s important to remember that the explanatory variables should be independent of each other.

First, let’s create some dummy data for our milkshake store.

(Video) OLS Simple Linear Regression _ Python

I’ve created three variables to help us predict our milkshake sales, temperature, the probability of raining on that day, and the daily closing value of Microsoft’s stock price.

Maybe, the last one is not the best option to help us predict milkshake sales.

There are many ways to check if a variable helps predict the outcome of another. Those methods are called Feature Selection.

One of the most used ways is checking the p-value.

The p-value tells us the probability of finding values as extreme as the ones observed in our tests— So, the coefficients are a measure of the relationship between two variables, and the p-value of the coefficients are a measure of the statistical significance of that relationship. Phew… sounds complicated, let’s see how to apply it.

We used Scikit’s class f_regression from feature_selection. This method returns the F values and the p-values, for now, we’ll just go over the p-values.

We’re looking for relationships with a statistical significance equal to or higher than 95%, in other words, a p-value lower than 5%.

As we can see from our results, Microsoft’s stock price is not a useful variable for predicting our milkshake sales.

Now, let’s finish this!

Cool!

Now we know that if the forecast is telling the temperature tomorrow will be 27 with only a 10% chance of rain, we expect to sell something around 171 milkshakes.

Sales = -33.91 + (-20.3 * 0.1) + (7.66 * 27)
Sales = -33.91 -2.03 +206.82
Sales = ~171

(Video) Multiple Linear Regression in Python - sklearn

And that’s it! There are lots of other things surrounding this topic, different methods of feature selection, metrics, assumptions to help you check your data, cross-validation methods, and so much more. I hope you could get an understanding of the basics, and that you’ll be able to build more knowledge on that.

Thanks for taking the time to read my article!

Did you know that we have four publications and a YouTube channel? You can find all of this from our homepage at plainenglish.io — show some love by giving our publications a follow and subscribing to our YouTube channel!