Introduction
Probability provides the mathematical foundation for statistical inference — every hypothesis test and confidence interval in later chapters rests on the probability concepts developed here.
1. Sample spaces and events
A sample space is the set of all possible outcomes of a random experiment; an event is any subset of the sample space (a specific outcome or combination of outcomes of interest). Correctly and completely defining the sample space is often the most important step in solving a probability problem, since a miscounted or incomplete sample space invalidates every subsequent calculation.
2. Axioms and rules of probability
Probability assigns a number between 0 and 1 to every event, following three basic axioms: every probability is non-negative, the probability of the entire sample space is 1 (something must happen), and for mutually exclusive events, the probability of their union is the sum of their individual probabilities. Every other probability rule can be derived from these three axioms.
3. Addition and multiplication rules
The addition rule, P(A∪B) = P(A) + P(B) − P(A∩B), gives the probability that at least one of two events occurs, subtracting the overlap (P(A∩B)) once to avoid double-counting it — for mutually exclusive events (no overlap), this simplifies to a direct sum. The multiplication rule, P(A∩B) = P(A)·P(B|A), gives the probability that both events occur, and simplifies to P(A)·P(B) specifically when A and B are independent (below).
4. Conditional probability and independence
Conditional probability, P(B|A) = P(A∩B)/P(A), is the probability of B given that A is already known to have occurred — it updates the probability of B based on the additional information that A happened. Two events are independent if knowing one occurred does not change the probability of the other, P(B|A) = P(B); independence must be checked, not assumed, and should not be confused with mutual exclusivity (which actually implies strong dependence — if mutually exclusive events cannot both occur, then knowing one happened makes the other’s probability exactly zero).
5. Bayes’ theorem
Bayes’ theorem, P(A|B) = P(B|A)P(A)/P(B), reverses the direction of a conditional probability — allowing you to find P(A|B) when only P(B|A) (and the individual probabilities) are known. This is especially useful in diagnostic and classification problems (for example, finding the probability a patient has a disease given a positive test result, when what is actually known from clinical data is the probability of a positive test given the disease is present or absent) and frequently produces results that are counter-intuitive without careful calculation, particularly when the underlying event is rare.
6. Random variables and expectation
A random variable assigns a numerical value to each outcome in a sample space, converting qualitative or descriptive outcomes into a form suitable for further mathematical analysis (developed fully in Chapter 3). The expectation (expected value) of a random variable is its probability-weighted average — the long-run average value it would take over many repetitions, which need not itself be a value the variable can actually take on any single trial (e.g., the expected value of a fair die roll is 3.5, though no single roll ever gives 3.5).
Key Takeaways
- Correctly defining the sample space is the essential first step; probability rules follow from three basic axioms (non-negativity, total probability 1, additivity for mutually exclusive events).
- The addition rule subtracts overlap to avoid double-counting; the multiplication rule requires conditional probability unless independence holds.
- Independence (knowing one event doesn’t change the other’s probability) is distinct from — and effectively opposite to — mutual exclusivity, which implies strong dependence.
- Bayes’ theorem reverses a known conditional probability to find the reverse conditional probability, essential for diagnostic and classification reasoning.
- A random variable’s expectation is its probability-weighted long-run average, not necessarily an achievable single-trial value.