Maths › Further Statistics 2 › Least squares regression and residuals
Least squares regression and residuals
A line of best fit chosen by a rule rather than by eye. The rule is to make the vertical gaps as small as possible in total, and the gaps left over tell you whether a line was the right idea.
Builds on Correlation and regression and Representing and interpreting data.
IN THIS TOPIC
- Calculate regression coefficients from summary statistics and write the line down.
- Compute residuals and use them to judge a fit and spot outliers.
- Find the residual sum of squares and say what it measures.
WHAT YOU PROBABLY THINK
The least squares line is the one that passes closest to all the points, measuring distance perpendicular to the line.
Minimising the vertical gaps
For each data point the residual is the vertical gap between the observed y and the value the line predicts: y − (a + bx). The least squares line is the one that makes the sum of the squares of those residuals as small as possible. The distances measured are vertical, not perpendicular, so the opening claim describes a different line altogether.
That choice is what makes the regression of y on x different from the regression of x on y: each minimises gaps in its own direction. Squaring, rather than taking the size, keeps the algebra differentiable and punishes one large miss more than several small ones.
WORKED EXAMPLE
A line from summary statistics
For the five points (1, 2), (2, 4), (3, 5), (4, 4), (5, 6), find the regression line of y on x.
Σx = 15, Σy = 21, Σx² = 55, Σxy = 71, so Sxx = 55 − 15²/5 = 10 and Sxy = 71 − (15)(21)/5 = 8.
b = 8/10 = 0.8, and a = 4.2 − 0.8 × 3 = 1.8.
The line is y = 1.8 + 0.8x. Predicted values are 2.6, 3.4, 4.2, 5.0, 5.8.
What the leftovers tell you
Residuals are worth plotting. If a straight line is the right model they scatter about zero with no pattern; a curve in the residual plot means the relationship was not linear, and one residual far larger than the rest flags a possible outlier or a recording error. The residuals always sum to zero, so their size, not their total, is what carries information.
The residual sum of squares measures what the line failed to explain:
Since Syy is the total variation in y, the subtracted term is the part the line accounts for. A small RSS relative to Syy means the line has done most of the work.
YOUR TURN
Residuals and RSS
For the same five points, find the residuals and the residual sum of squares, and comment.
Show the working
Observed minus predicted: −0.6, 0.6, 0.8, −1.0, 0.2. They sum to zero, as they always do.
Sum of squares: 0.36 + 0.36 + 0.64 + 1.00 + 0.04 = 2.4.
By the formula: Syy = 97 − 21²/5 = 8.8, so RSS = 8.8 − 64/10 = 2.4, which agrees.
The line explains 6.4 of the 8.8 total variation. The largest residual, −1.0 at x = 4, is not extreme enough to call an outlier.
THE EXAM BIT
- Quote the formulae for the coefficients and show the summary statistics you put into them.
- Residual means observed minus predicted; the other order is the commonest sign error.
- Interpret a residual plot in words: a pattern rules the model out, scatter supports it.
- For the RSS, either sum the squared residuals or use the formula, and say which you used.
CHECK YOURSELF
A regression line gives Sxx = 20, Sxy = 30 and Syy = 50. Find the gradient and the residual sum of squares.
Show a hint
One division for the gradient, then the RSS formula.
Show the answer
b
=
3
0
/
2
0
=
1
.
5
.
R
S
S
=
5
0
−
9
0
0
/
2
0
=
5
0
−
4
5
=
5
,
s
o
t
h
e
l
i
n
e
e
x
p
l
a
i
n
s
4
5
o
f
t
h
e
5
0
t
o
t
a
l
v
a
r
i
a
t
i
o
n
.
Least squares minimises the sum of the squared vertical residuals, giving b = S(xy)/S(xx), with the line through the mean point.
Residuals sum to zero; their pattern tests the model and RSS = S(yy) − S(xy)²/S(xx) measures what is left unexplained.
WORKBOOK
Printable practice for this topic: original exam-style questions with room to work, and a fully worked answer book. Free to use; please do not redistribute or sell.
CHECK YOUR PROGRESS
Rate how confident you feel with each objective for this lesson. Ratings are saved in this browser, on this device only.
- Calculate regression coefficients from summary statistics and write the line down.
- Compute residuals and use them to judge a fit and spot outliers.
- Find the residual sum of squares and say what it measures.
Open the full revision checklist to see every objective in the course in one place.
No animated video for this topic yet; these notes stand alone.