Introduction
Regression and correlation quantify relationships between two (or more) variables — moving from the single-variable summaries of Chapter 1 to describing how variables move together, and forming the foundation of predictive statistical modelling.
1. Scatter diagrams
A scatter diagram plots paired observations of two variables against each other, providing an immediate visual impression of whether a relationship exists, its rough form (linear or otherwise), its direction (positive or negative), and its strength — this visual step should always precede any numerical correlation or regression calculation, since it can reveal patterns (such as clear non-linearity, or the influence of outliers) that summary numbers alone can obscure or misrepresent.
2. Correlation coefficient and interpretation
The Pearson correlation coefficient, r, measures the strength and direction of a linear relationship between two variables, ranging from −1 (perfect negative linear relationship) through 0 (no linear relationship) to +1 (perfect positive linear relationship). Critically, r measures only linear association — two variables can have a strong, obvious non-linear relationship (like a perfect curve) while still producing a correlation coefficient near zero, which is exactly why the scatter diagram of point 1 must always be checked first.
3. Simple linear regression
Simple linear regression fits a straight line, ŷ = a + bx, to describe how a response variable y depends on a single explanatory variable x, allowing prediction of y for a given x value. Unlike correlation (which is symmetric between the two variables), regression is directional — it specifically models y as depending on x, and swapping which variable is treated as the response generally produces a different line.
4. The least-squares idea
The regression line’s coefficients (a, b) are chosen by the method of least squares: minimising the sum of squared vertical distances between each observed data point and the fitted line (the residuals). Squaring the residuals before summing serves two purposes: it prevents positive and negative residuals from cancelling each other out, and it penalises larger deviations disproportionately more than smaller ones, so the fitted line is pulled toward minimising the largest errors especially.
5. Regression coefficients and prediction
The slope (b) represents the predicted change in y for a one-unit increase in x, expressed in the actual units of both variables (essential for meaningful interpretation — a slope of 2 means something entirely different depending on whether x is measured in years or seconds). The intercept (a) is the predicted value of y when x = 0, though this value is only practically meaningful if x = 0 falls within, or reasonably near, the range of the actually observed data.
6. Limits of association and the distinction from causation
A regression or correlation result describes only statistical association between two variables — it does not, by itself, establish that changes in one variable actually cause changes in the other. A strong relationship may instead arise from a confounding variable influencing both, from reverse causation, or from pure coincidence, particularly in small samples. Similarly, extrapolating a fitted regression line far beyond the range of the observed data assumes the same linear relationship continues to hold outside that range — an assumption that frequently fails in practice and should always be flagged as a limitation when a question invites prediction outside the observed range.
Key Takeaways
- A scatter diagram should always be examined before calculating correlation or regression, since it reveals patterns (non-linearity, outliers) that summary statistics can miss or misrepresent.
- The Pearson correlation coefficient measures only linear association; a near-zero value does not rule out a strong non-linear relationship.
- Regression is directional (models y depending on x) while correlation is symmetric; the least-squares method fits the line by minimising the sum of squared residuals.
- The slope and intercept must be interpreted in the actual units and range of the data; the intercept is meaningful only near x = 0 if data exists there.
- Statistical association (correlation/regression) never by itself establishes causation, and extrapolating beyond the observed data range is a specific, frequently tested limitation.