Sampling distributions and the CLT

Population, sample and sampling distribution

A population is every unit a question is about: every cup a coffee machine will pour, every bug a team will fix. A sample is the few units actually measured. A statistic such as the sample mean \(\bar{x}\) is computed from the sample, so it changes from one sample to the next. Its sampling distribution is the distribution of those values over all the samples that could have been drawn.

Three distributions are easily blurred, and most mistakes with this chapter's ideas come from blurring them:

The standard error

If the n values are drawn independently from a population with mean \(\mu\) and standard deviation \(\sigma\), the sample mean is centred on \(\mu\), and its spread shrinks with the square root of n:

That spread is the standard error of the mean. The standard deviation \(\sigma\) says how much single values vary, and more data does not change it; the standard error says how much the estimate would vary from sample to sample, and more data shrinks it. Bug fixes that take anywhere from 1 to 5 hours have \(\sigma = \tfrac{4}{\sqrt{12}} = 1.1547\) hours. The mean of 16 fixes has a standard error of 0.2887 hours, and the mean of 64 fixes 0.1443. Four times the data halves the standard error, so precision comes at a rising price.

The central limit theorem

The standard error gives the spread of \(\bar{X}\). The central limit theorem (CLT) adds its shape: for independent draws from almost any population with a finite standard deviation, the sample mean is approximately normal once n is large enough, whatever the population looks like:

Three promises come bundled: the centre is \(\mu\), the spread is \(\tfrac{\sigma}{\sqrt{n}}\), and the shape becomes normal. The first two hold exactly for every n; only the shape needs n to be large.

Waiting times with a mean of 3 minutes are exponential: strongly skewed to the right, with \(\sigma = 3\). Draw 2,000 samples of 5 and the histogram of their means is still skewed. Its standard deviation is 1.377, against the predicted \(\tfrac{3}{\sqrt{5}} = 1.3416\).

With samples of 30 the means are much closer to the bell. Their standard deviation is 0.5547 against the predicted 0.5477, and on the same axis the histogram is far narrower.

How large must n be?

The usual rule is \(n \ge 30\). It is a rule of thumb, not a law: what matters is how far the population is from normal.

• A normal population. The mean of normal values is exactly normal for every n, even n = 2.
• A symmetric population, such as the uniform or the bimodal, is close to normal quickly: by n = 10 the means look like a bell.
• A skewed population needs more. The skewness of the mean is the population's divided by \(\sqrt{n}\). For the exponential, whose skewness is 2, that is 0.8944 at n = 5 and still 0.3651 at n = 30.

The tails settle last. For the mean of 30 exponential waits with a mean of 3, the normal approximation gives \(P(\bar{X} > 4.5) = 0.0031\). The exact answer, from the gamma distribution the mean really follows, is 0.0073: more than twice as large. For a question about the middle of the distribution, n = 30 is plenty; for a question about a rare event, it may not be.

What the CLT does not say

• The data do not become normal. Ten thousand exponential waiting times draw a detailed exponential, not a bell. Only the distribution of the mean changes shape.
• It is about the mean, not single values. It says nothing about the share of customers above a threshold; that needs the population's own distribution.
• It does not repair a biased sample. If the sample is not random, its mean is centred on the wrong value, however large n is.
• It needs independence and a finite \(\sigma\). Values that move together, or a population with tails so heavy that \(\sigma\) is infinite, such as Student's t with \(\nu = 1\), break it: the mean of a million such values is no more settled than a single one.

n and the number of samples

The simulator has two numbers that are easily confused. n, the size of each sample, is the CLT's knob: it narrows the sampling distribution and makes it normal. The number of samples only decides how smoothly the histogram shows that distribution: 100 samples give a ragged picture and 2,000 a smooth one, but its width is \(\tfrac{\sigma}{\sqrt{n}}\) either way.

In practice there is only one sample. The sampling distribution is the thought experiment of drawing it again and again, and the standard error \(\tfrac{s}{\sqrt{n}}\), with the sample's own s, estimates its spread from the one sample at hand. Chapter 11 builds on exactly that.

Annualising returns: the square root of time

The same arithmetic scales risk over time. A year's log return is the sum of about 250 daily log returns. If the days are independent, each with mean \(\mu\) and standard deviation \(\sigma\), the sum over T days has

The simulated share's daily log returns in the lab's data have \(\mu = 0.0012\) and \(\sigma = 0.0279\), so a year of 250 trading days has a mean of 0.3072 and a standard deviation of \(\sqrt{250}\) times the daily one, 0.4412, as chapter 7 used. A week of 5 days has a standard deviation of 0.0624, not five times the daily figure. Risk grows with the square root of time, the mean with time itself. By the CLT, a sum of many daily returns is also closer to normal than the returns themselves, provided the days are independent; share prices break that in a crisis, when volatile days cluster.

The standard error shows how little a year of prices says about the average return. The mean daily return, 0.0012, has a standard error of \(\tfrac{0.0279}{\sqrt{251}} = 0.0018\): the estimate is less than one standard error from zero. A year of data measures the volatility well and the average return hardly at all.

In code

Excel's RAND() gives one draw per cell: copy the formula into 30 columns and 2,000 rows, then average each row. STDEV.S and numpy's std(ddof=1) divide by n − 1; chapter 11 explains why.