Student's t and confidence intervals

Point estimates

An estimator is a rule, such as "average the sample"; an estimate is the number the rule gives for the sample at hand. The sample mean \(\bar{x}\) estimates the population mean \(\mu\), the sample standard deviation \(s\) estimates \(\sigma\), and the sample proportion \(\hat{p}\) estimates a probability \(p\). Because the sample is random, the estimator is a random variable, and its distribution is chapter 10's sampling distribution.

A regional fire service records the damage from 10 wildfires (the lab's dataset wildfire-damages.csv), in thousands of dollars: 15, 80, 31, 104, 131, 46, 15, 62, 16 and 36. The point estimate of the mean damage per fire is \(\bar{x} = 53.6\) thousand dollars. On its own that number says nothing about how far off it may be: with 10 fires that vary this much, the true mean could well be 30 or 80.

Bias and Bessel's n − 1

An estimator is unbiased if, averaged over all possible samples, it hits the true value; \(\bar{x}\) is unbiased for \(\mu\). The spread is subtler. The deviations are measured from the sample's own centre \(\bar{x}\), which sits closer to the data than \(\mu\) does, so they are too small on average. Dividing their squares by n underestimates \(\sigma^2\) by a factor of \(\tfrac{n - 1}{n}\); dividing by \(n - 1\) instead, Bessel's correction, removes the bias:

For the wildfires, \(s = 40.24\) thousand dollars with \(n - 1 = 9\); dividing by 10 instead gives 38.17, and on average that version's variance is only 0.9 of the true one. The difference matters for small samples and fades for large ones: at n = 100 the factor is 0.99. The missing degree of freedom went into estimating \(\bar{x}\).

The t statistic

Standardising \(\bar{x}\) with a known \(\sigma\) gives a standard normal z. With \(\sigma\) unknown, its estimate \(s\) takes its place, and \(s\) varies from sample to sample too. The result is no longer normal: it follows Student's t distribution with \(\nu = n - 1\) degrees of freedom, provided the population is roughly normal:

The t is centred on 0 and symmetric like the standard normal, but its tails are heavier, because a sample that happens to have a small s pushes t far out. The chance of landing more than 2 from 0 is 0.1835 with 2 degrees of freedom, 0.0766 with 9 and 0.0511 with 49, against 0.0455 for the normal. As \(\nu\) grows, s settles on \(\sigma\) and the t converges to the normal.

The t* table

For an interval, the value that matters is \(t^*\), which leaves \(\tfrac{1 - C}{2}\) in each tail for a confidence level C. It is larger than the normal's \(z^*\): by a lot for small samples, and hardly at all for large ones.

With 49 degrees of freedom, as for 50 cups of coffee, \(t^* = 2.0096\) for 95 %, against \(z^* = 1.96\); with 9, as for 10 wildfires, it is 2.2622. The 1.96 of every textbook is the bottom row.

z and t intervals

A confidence interval puts a margin around the estimate, wide enough to catch the true mean with a chosen probability. Every interval here has the same anatomy:

With \(\sigma\) known, the critical value is the normal's \(z^*\); with \(\sigma\) estimated by s, it is \(t^*\) with \(n - 1\) degrees of freedom:

The t interval assumes a roughly normal population when n is small; for large n the CLT covers other shapes. A known σ is rare in practice: it fits a long-running process whose spread is well known while its mean may drift, such as a machine.

Coffee caffeine

A café's machine pours black coffee whose caffeine varies from cup to cup with a standard deviation long known to be 7 mg. The lab's sample of 50 cups (coffee-caffeine.csv) has a mean of 113.18 mg and a standard deviation of 8.01 mg. With \(\sigma = 7\) taken as known, the 95 % z interval is

Without trusting the old \(\sigma\), the t interval uses the sample's \(s = 8.01\) and \(t^* = 2.0096\): \(113.18 \pm 2.0096 \times \tfrac{8.01}{\sqrt{50}} = 113.18 \pm 2.28\), from 110.91 to 115.46 mg. It is wider for two reasons: this sample's s happens to be larger than 7, and \(t^*\) is larger than \(z^*\).

Wildfire damages

For the 10 wildfires, \(\bar{x} = 53.6\), \(s = 40.24\) and \(t^* = 2.2622\) with 9 degrees of freedom, so the 95 % interval for the mean damage is \(53.6 \pm 2.2622 \times \tfrac{40.24}{\sqrt{10}} = 53.6 \pm 28.78\), from 24.82 to 82.38 thousand dollars. The interval is wide, and honest about it: 10 fires that vary this much pin the mean down only roughly.

It also leans on an assumption the data strain. Damages are skewed to the right, a few costly fires among many modest ones, and with n = 10 the CLT has not yet made the mean normal, so this interval may cover the true mean less often than 95 % of the time. With a sample this small and this skewed, report the interval with that caveat, or collect more fires.

The three levers

Three things set the width of an interval: the confidence level, the sample size and the spread. For the coffee machine with \(\sigma = 7\):

The trade-off cannot be escaped: the only way to be more confident without more data is to say something vaguer. A 100 % interval for the mean caffeine would run from nothing to the most a cup can hold: certain, and useless.

Reading an interval correctly

A 95 % interval from 111.24 to 115.12 mg does not mean there is a 95 % probability that the machine's mean lies between those numbers. The mean is a fixed number, inside the interval or not; what is random is the interval, which another sample would move. The 95 % is a property of the procedure: of all the intervals built this way from new samples, 95 % contain the true mean.

Nor does it say that 95 % of cups lie in the interval. It is an interval for the mean, and single cups vary far more: with σ = 7, a cup lies within 1.94 mg of the mean only about 22 % of the time. Questions about single values need the population's distribution, not the mean's.

The coverage simulation makes the procedure visible. It draws 100 samples from a machine whose true mean it knows, builds an interval from each and marks the ones that miss. About 95 of the 100 contain the mean, and which ones miss is chance. Using \(z^*\) with the sample's s when n is small is the classic slip: with n = 5 that covers the mean only about 88 % of the time, where t covers 95 %.

From intervals to tests

An interval is a range of plausible values for the mean, which makes it half of a test already: a claimed mean outside the 95 % interval is one the data would reject in a two-sided test at the 5 % level. If the machine's maker claims 110 mg a cup, the z interval from 111.24 to 115.12 mg says the claim is implausible, and the t interval, from 110.91, says the same by a narrower margin. How to set up such a test, the p-value, and the two errors a test can make are the subject of the Hypothesis Testing Lab, the next lab in this series.

In code

Excel's CONFIDENCE.NORM and CONFIDENCE.T return the margin of error, not the interval: subtract it from the mean and add it to the mean. Beware numpy: x.std() divides by n unless ddof=1 is given; pandas' std() divides by n − 1.