What are hypothesis testing and ANOVA
Hypothesis testing is the statistical method for deciding, with a known risk of error, whether a difference seen in the data is real or just chance. ANOVA (analysis of variance) is the hypothesis test used to compare the means of three or more groups at once, such as three machines, three suppliers or three shifts.
In process improvement, this is the step that separates opinion from evidence: before adjusting a machine or changing a supplier, the test tells you whether the difference you saw in the samples justifies action. It is a tool of the Analyze phase of Six Sigma DMAIC.
Hypothesis testing visual map
The map shows the test flow (hypothesis, p-value, decision) and the three-machine ANOVA example solved in this article.
The concepts in 5 lines
- H0 (null hypothesis): there is no difference (the means are equal). It is the starting point.
- H1 (alternative hypothesis): there is a difference (at least one mean differs).
- Significance level α: the type I error risk you accept, usually 0.05.
- p-value: the probability of seeing a difference as large as the one obtained, or larger, if H0 were true.
- Decision rule: if p < α, reject H0; if p ≥ α, there is no evidence to reject it.
| Reality \ Decision | Reject H0 | Fail to reject H0 |
|---|---|---|
| H0 true | Type I error (false alarm), probability α | Correct decision |
| H0 false | Correct decision (power, 1 − β) | Type II error (real difference missed), probability β |
The test flow in 6 steps
- State H0 and H1 in process terms (for example, "mean fill weight is the same on all three machines").
- Choose α before looking at the data.
- Set the sample size and collect the data randomly, with a trustworthy measurement system.
- Check the assumptions: independence, approximate normality and similar variances.
- Calculate the test statistic (t or F) and the p-value.
- Decide and translate it into a process action, looking at the size of the difference as well, not only the p-value.
Comparing two groups: the t test
Illustrative example: a filling line for 50 g packs has three machines (A, B and C). Five packs from each were weighed.
| Pack | Machine A | Machine B | Machine C |
|---|---|---|---|
| 1 | 50.2 | 50.6 | 50.1 |
| 2 | 49.8 | 50.9 | 50.3 |
| 3 | 50.5 | 50.4 | 49.9 |
| 4 | 50.1 | 51.0 | 50.2 |
| 5 | 49.9 | 50.7 | 50.0 |
| Mean (g) | 50.10 | 50.72 | 50.10 |
| Variance | 0.075 | 0.057 | 0.025 |
First, compare only A and B with the equal-variance t test. The pooled variance is sp² = (0.075 + 0.057) ÷ 2 = 0.066, and the standard error of the difference is 0.1625.
t = (50.10 − 50.72) ÷ 0.1625 = -3.82
With 8 degrees of freedom, the two-sided critical t at 5% is 2.306. Since |-3.82| > 2.306 (p = 0.0051), H0 is rejected: machines A and B fill at different means.
One-way ANOVA: three machines at once
With three groups, comparing pairs with several t tests inflates the risk: three comparisons at 5% give a false-alarm chance of 1 − 0.95³ = 14.3%; with 5 groups (10 pairs) it climbs to 40.1%. ANOVA makes a single overall comparison.
F = MSbetween ÷ MSwithin = [SSbetween ÷ (k − 1)] ÷ [SSwithin ÷ (N − k)]
SSbetween measures how far the group means sit from the grand mean; SSwithin measures the variation inside each group (the noise). If the signal beats the noise by a wide margin, F is large.
Calculation with the data above (k = 3 groups, n = 5 per group, N = 15). Grand mean = 50.31 g.
- SSbetween = 5 × [(50.10 − 50.31)² + (50.72 − 50.31)² + (50.10 − 50.31)²] = 1.281
- SSwithin = sum of squared deviations inside each group = 0.628
- MSbetween = 1.281 ÷ 2 = 0.6407; MSwithin = 0.628 ÷ 12 = 0.0523
F = 0.6407 ÷ 0.0523 = 12.24
| Source | SQ | gl | MQ | F | p |
|---|---|---|---|---|---|
| Between machines | 1.281 | 2 | 0.6407 | 12.24 | 0.0013 |
| Within (error) | 0.628 | 12 | 0.0523 | ||
| Total | 1.909 | 14 |
The critical F at 5% with 2 and 12 degrees of freedom is 3.885. Since 12.24 > 3.885, and p = 0.0013 < 0.05, H0 is rejected: at least one machine has a different mean.
Which machine differs? Tukey's post hoc test
ANOVA says a difference exists but not where. Tukey's test compares all pairs while holding the overall risk at 5%. The minimum significant difference is HSD = q × √(MSwithin ÷ n), with q = 3.773 (3 groups, 12 df).
HSD = 3.773 × √(0.0523 ÷ 5) = 0.386 g
| Pair | Difference of means (g) | Versus HSD 0.386 | Conclusion |
|---|---|---|---|
| B − A | 0.62 | larger | Differ |
| B − C | 0.62 | larger | Differ |
| A − C | 0.00 | smaller | Do not differ |
Process conclusion: machine B fills about 0.62 g more than A and C (1.2% of the 50 g nominal weight). Adjust or investigate B (nozzle, pressure, calibration) and confirm with a new sample. To follow up, use a control chart.
Common mistakes and assumptions
- Confusing "fail to reject H0" with "H0 is true": with a small sample, the test may simply lack the power to detect the difference.
- Treating p = 0.049 and p = 0.051 as opposite worlds: the p-value is a continuous measure of evidence.
- Ignoring practical significance: with huge samples, irrelevant differences become "significant". Compare the difference with the product tolerance, for example through Cp and Cpk.
- Not checking the assumptions: ANOVA assumes independent observations, roughly normal residuals and similar variances across groups. Use the histogram and a residual plot; if they fail, consider a non-parametric test such as Kruskal-Wallis.
- Running many tests and reporting only the one that worked: each extra test raises the false-positive risk.
- Non-random samples: pulling parts only at the start of the shift distorts the result.
To relate two continuous variables instead of comparing groups, see the scatter diagram and correlation. To express the result in defects per million, see the sigma level.
Frequently asked questions
What is a p-value in plain language?
It is the probability of seeing a difference as large as the one observed if there were actually no difference at all. The smaller it is, the harder it is to blame chance.
When should I use ANOVA instead of a t test?
When there are three or more groups. With two groups, ANOVA and the t test give the same result (F = t²).
What does p < 0.05 mean?
That if H0 were true, a difference like this would appear less than 5% of the time. By convention, H0 is rejected.
In the example, why is F = 12.24 significant?
Because it exceeds the critical F of 3.885 (5%, 2 and 12 df); the matching p-value is 0.0013.
What do I do after a significant ANOVA?
Run a post hoc test such as Tukey to see which groups differ, then check whether the difference matters for the product.
How many samples do I need?
It depends on the difference worth detecting and on process variation. As a starting point use at least 5 to 10 per group and calculate the test power in statistical software.
Sources
- MONTGOMERY, D. C.; RUNGER, G. C. Applied Statistics and Probability for Engineers. Wiley.
- MONTGOMERY, D. C. Design and Analysis of Experiments. Wiley.
- FISHER, R. A. Statistical Methods for Research Workers. Edinburgh: Oliver and Boyd, 1925.
- TUKEY, J. W. Comparing individual means in the analysis of variance. Biometrics, v. 5, 1949.
- PYZDEK, T.; KELLER, P. The Six Sigma Handbook. McGraw-Hill.
Keep studying: trusted sources and links
Save this address: Manufatura Enxuta is your school for lean manufacturing. Come back whenever you need to study, with a complete article, a visual map and trusted sources. Follow one of the learning paths or check the A–Z glossary.
Want to measure what really matters?
Download the free e-book on performance indicators and track the losses in your operation.
Download the e-book
About the author
Lean Manufacturing & Behavioral Management Specialist
Over 20 years in the automotive and metalworking industries (GM and Dana), Lean Manufacturing practitioner since 2006. SENAI instructor and mentor in Brazil’s Brasil Mais Produtivo program, delivering consulting, training and audits for 50+ companies, combining quality, productivity and people development.
