If the p-value is 0.03, is there a 3 % chance that the null hypothesis is true?

No. The p-value is computed assuming H₀ is true: it is the probability of data at least as extreme as yours in that hypothetical world, P(data this extreme | H₀). The probability that H₀ is true given the data, P(H₀ | data), would also depend on how plausible H₀ was before the experiment, which a p-value does not use at all. The simulator makes this visible: every one of the 1000 simulated experiments comes from a world where H₀ is exactly true, and about 3 % of them still land beyond z_obs. A small p says the data are unusual under H₀, not that H₀ is unlikely.

Why does the same observed mean give a smaller p-value when n is larger?

Because the standard error σ/√n shrinks with n. With n = 25 and σ = 15 a class mean of 104 is only 1.33 standard errors above 100, which happens by chance about 18 % of the time in a two-sided test; with n = 100 the same 4-point difference is 2.67 standard errors and p drops to about 0.008. The raw effect is identical, but a mean of many observations fluctuates less under H₀, so the same deviation becomes harder to explain by chance. This is why statistical significance alone does not measure how large or important an effect is.

What is the difference between a one-tailed and a two-tailed p-value?

A two-tailed test asks how often H₀ produces a statistic at least as far from μ₀ in either direction, so p = 2·Φ(−|z|) and the shaded area sits in both tails. A one-tailed test specifies the direction in advance — for example that the drug lowers blood pressure — and only counts deviations that way, giving p = 1 − Φ(z) for a right-tailed alternative. For the same z the one-tailed p is exactly half the two-tailed one, and the critical value moves from ±1.96 to 1.645 at α = 0.05. The direction must be chosen before looking at the data; picking the tail that happens to fit afterwards doubles the real false-positive rate.

Why does the coin example use a normal curve when the number of heads is binomial?

The number of heads in n fair flips is Binomial(n, 0.5), a discrete distribution, but for large n it is closely approximated by a normal curve with mean np₀ and standard deviation √(np₀(1−p₀)); in terms of the proportion p̂ = heads/n that is mean p₀ and SE = √(p₀(1−p₀)/n). With n = 100 the approximation is good: 60 heads gives z = 2.0 and a two-sided p of 0.046, while the exact binomial two-sided probability of 60 or more (or 40 or fewer) heads is 0.057. The simulated experiments flip real coins, so their histogram is stepped and the empirical fraction follows the exact binomial rather than the smooth curve. For small n or a p₀ far from 0.5 the approximation worsens and the exact binomial test should be used instead.

p-Value Explained

Test statistic on the null density; shaded one- or two-sided tail is the p-value. Why it is not the probability that H₀ is true.

Hypothesis test

Scenario

Alternative hypothesis (tail)

Significance level α

Sample size n (same x̄, new SE)100

One-sample z-test with σ known: under H₀ the sample mean is normal with mean μ₀ and standard error σ/√n, so z = (x̄ − μ₀)/SE is standard normal. The coin scenario is a proportion test using the normal approximation to the binomial with SE = √(p₀(1−p₀)/n) and no continuity correction; its simulated experiments really flip n coins, so their histogram is discrete. Dragging the marker changes the observed value; changing n keeps x̄ fixed and clears the simulated experiments.

Shortcuts

Space / Enter — simulate 1000 experiments under H₀
R — reset the scenario (new random seed)
Drag the pink marker — move the observed statistic

Measured values

Null value μ₀ (H₀)0.500

σ = √(p₀(1−p₀))0.500

Sample size n100

Observed p̂0.600 (≈ 60 of 100 heads)

SE = σ/√n0.05000

z_obs = (x̄ − μ₀)/SE2.000

p-value (exact)0.0455

Empirical fraction ≥ as extreme—

Significance level α0.05

Critical z±1.960

Verdictreject H₀

About this model

The simulator shows a one-sample z-test with a known population standard deviation σ. Under the null hypothesis H₀: μ = μ₀ the sample mean of n observations is normally distributed with mean μ₀ and standard error SE = σ/√n, so the test statistic z = (x̄ − μ₀)/SE follows the standard normal distribution. The canvas draws that standard normal density in z units with a second axis in the raw units of the scenario (fraction of heads, IQ points, mmHg), the observed statistic z_obs as a draggable marker, and the p-value as the shaded tail: p = 1 − Φ(z) for a right-tailed alternative, Φ(z) for a left-tailed one and 2·Φ(−|z|) for a two-tailed one, where Φ is the standard normal cumulative distribution function. Φ is computed with Cody's rational Chebyshev approximation of erfc (error far below 1e-7) and the critical values z_c = Φ⁻¹(1 − α) or ±Φ⁻¹(1 − α/2) with Acklam's inverse followed by a Newton step, so the verdict 'reject H₀' (p < α) is exact and not read off a table. The coin scenario is a test of a proportion, H₀: p = 0.5, using the normal approximation to the binomial with SE = √(p₀(1−p₀)/n) and no continuity correction; when you simulate experiments in that scenario the program really flips n coins per experiment, so the histogram is discrete and its tail fraction can differ slightly from the smooth approximation. The empirical layer draws 1000 experiments from H₀ itself with a seeded generator, plots their statistics as a histogram under the curve and counts the fraction at least as extreme as z_obs; that fraction is a Monte-Carlo estimate of the p-value and makes its meaning concrete — how often chance alone, under H₀, produces data this extreme. The sample-size slider keeps x̄ fixed and shrinks SE, which shows why the same raw effect becomes significant as n grows. Simplifications: σ is treated as known (otherwise a t-statistic would be needed), observations are assumed independent and identically distributed, and no correction for multiple testing is made. Related simulators: statistics/type-i-ii-errors-power (α, β and power as areas under two curves), statistics/confidence-interval (the dual of a two-sided test), statistics/normal-distribution (Φ and z-scores), statistics/central-limit-theorem (why the sampling distribution of x̄ is normal) and statistics/binomial-distribution (the exact distribution behind the coin scenario).

Who it's for: Upper secondary and first-year university students meeting hypothesis testing for the first time (AP Statistics, A-level, introductory statistics).

Key terms

null hypothesis H₀
test statistic z
sampling distribution
standard error σ/√n
p-value
significance level α
critical value
one- and two-tailed test

How it works

Under the null hypothesis H₀ the sample mean fluctuates around μ₀ with standard error σ/√n, so the test statistic z = (x̄ − μ₀)/SE follows a standard normal curve. The p-value is the shaded tail: the probability, if H₀ were true, of getting a statistic at least as far from μ₀ as the one observed — one tail for a directional alternative, both tails for a two-sided one. It is P(data this extreme | H₀), not P(H₀ | data): a p-value of 0.03 means such data arise in 3 % of worlds where H₀ holds, not that H₀ has a 3 % chance of being true. Pressing Simulate draws 1000 experiments from H₀ itself and counts how often they beat z_obs; that fraction converges to the exact p-value, which is what the p-value really measures. Sliding n keeps the same raw effect but shrinks SE, so the same x̄ becomes more surprising and p falls — the model assumes a known σ and a normal (or normal-approximated) sampling distribution.

Key equations

z = (x̄ − μ₀) / SE, SE = σ / √n (coin: SE = √(p₀(1−p₀)/n))

p = 1 − Φ(z) (right) · Φ(z) (left) · 2·Φ(−|z|) (two-tailed)

reject H₀ ⇔ p < α ⇔ z beyond z_c = Φ⁻¹(1 − α) or ±Φ⁻¹(1 − α/2)

Φ(z) = ½·erfc(−z/√2)

Frequently asked questions

If the p-value is 0.03, is there a 3 % chance that the null hypothesis is true?: No. The p-value is computed assuming H₀ is true: it is the probability of data at least as extreme as yours in that hypothetical world, P(data this extreme | H₀). The probability that H₀ is true given the data, P(H₀ | data), would also depend on how plausible H₀ was before the experiment, which a p-value does not use at all. The simulator makes this visible: every one of the 1000 simulated experiments comes from a world where H₀ is exactly true, and about 3 % of them still land beyond z_obs. A small p says the data are unusual under H₀, not that H₀ is unlikely.
Why does the same observed mean give a smaller p-value when n is larger?: Because the standard error σ/√n shrinks with n. With n = 25 and σ = 15 a class mean of 104 is only 1.33 standard errors above 100, which happens by chance about 18 % of the time in a two-sided test; with n = 100 the same 4-point difference is 2.67 standard errors and p drops to about 0.008. The raw effect is identical, but a mean of many observations fluctuates less under H₀, so the same deviation becomes harder to explain by chance. This is why statistical significance alone does not measure how large or important an effect is.
What is the difference between a one-tailed and a two-tailed p-value?: A two-tailed test asks how often H₀ produces a statistic at least as far from μ₀ in either direction, so p = 2·Φ(−|z|) and the shaded area sits in both tails. A one-tailed test specifies the direction in advance — for example that the drug lowers blood pressure — and only counts deviations that way, giving p = 1 − Φ(z) for a right-tailed alternative. For the same z the one-tailed p is exactly half the two-tailed one, and the critical value moves from ±1.96 to 1.645 at α = 0.05. The direction must be chosen before looking at the data; picking the tail that happens to fit afterwards doubles the real false-positive rate.
Why does the coin example use a normal curve when the number of heads is binomial?: The number of heads in n fair flips is Binomial(n, 0.5), a discrete distribution, but for large n it is closely approximated by a normal curve with mean np₀ and standard deviation √(np₀(1−p₀)); in terms of the proportion p̂ = heads/n that is mean p₀ and SE = √(p₀(1−p₀)/n). With n = 100 the approximation is good: 60 heads gives z = 2.0 and a two-sided p of 0.046, while the exact binomial two-sided probability of 60 or more (or 40 or fewer) heads is 0.057. The simulated experiments flip real coins, so their histogram is stepped and the empirical fraction follows the exact binomial rather than the smooth curve. For small n or a p₀ far from 0.5 the approximation worsens and the exact binomial test should be used instead.

Continue with similar topics in this category — or all 10 in Statistics & Probability.

View category →

NewSchool

Type I & II Errors and Statistical Power

Launch Simulator

Two normal curves (H₀ and H₁), a decision threshold; α, β and power 1−β as shaded areas; effect size, σ and n sliders.

NewSchool

Confidence Interval Simulator

Launch Simulator

Draw a sample, build a 90/95/99 % interval for μ; repeat 100 times and count how many intervals catch the true mean.

NewKids

Normal Distribution & z-Scores

Launch Simulator

Bell curve with μ and σ; shaded tails, the 68–95–99.7 rule, and a built-in z-score converter x → z = (x − μ)/σ.

NewFeaturedKids

Central Limit Theorem

Launch Simulator

Pick a parent (uniform, exponential, binomial, die), draw samples of size n = 1, 5, 30; the histogram of means turns normal and its spread shrinks as σ/√n.

NewKids

Law of Large Numbers

Launch Simulator

Coin or die: running mean x̄ₙ for several runs converges to μ; the ±σ/√n envelope shows how slowly.

NewFeaturedKids

Galton Board (Bean Machine)

Launch Simulator

Balls bounce left/right through rows of pegs; the bins fill a binomial that turns into a bell curve. Few rows vs many rows.