Math
The p-value, and how to fool yourself with it
Two thousand experiments ask whether a mean is zero; the share that rejects at α = 0.05 is counted. With no effect it is 5 % - the type I rate - and the p-values are uniform. With an effect it is the power, which rises with the effect and with n: half a σ is found 32 % of the time with ten observations and 82 % with thirty-four. Look at the data every five observations and stop when p dips below 0.05, and the false positive rate is a quarter. Everything is counted; Student's curve comes from the code.
What happens
A p-value answers a strange question: if there were no effect at all, how often would data this surprising turn up? This module runs the experiment two thousand times. With no effect, five per cent of runs cross the line - by construction, the p-value is uniform. With a real effect the fraction that crosses is the power, and it depends on the effect and on n: a small effect at n = 20 is caught one time in seven. Peek at the data as it comes and stop when p dips below 0.05, and the false-positive rate climbs past a quarter. Set δ and n, read the power, then try optional stopping.
Equations used in the module
Equations
t = m/(s/√n), Student with n − 1 degrees of freedom
under H₀ the p-value is uniform: P(p < α) = α
power ≈ Φ(δ√n − z_{1−α/2}) + Φ(−δ√n − z_{1−α/2})
optional stopping (look every 5, stop at p < 0.05): 27 % false positives
