What attribute agreement analysis is
Attribute agreement analysis (attribute MSA) measures whether inspectors classify parts as good or bad consistently with each other, over time, and against a known standard. The kappa statistic (Cohen) measures that agreement after removing the part that would occur by chance.
It applies when the inspection result is a decision (accept/reject, go/no-go, defect present/absent) rather than a measured value. For continuous data, the study is Gauge R&R.
Kappa and agreement visual map
The map shows the agreement matrix for two inspectors, the kappa calculation, and the effectiveness, miss rate and false alarm rate.
Cohen's kappa formula
Kappa = (po − pe) ÷ (1 − pe)
- po: observed proportion of agreement (parts where both give the same classification).
- pe: proportion of agreement expected by chance, computed from the row and column totals.
Kappa = 1 is perfect agreement; 0 is agreement equal to chance; negative values are worse than chance. Plain percent agreement misleads: two inspectors who pass almost everything agree a lot without seeing any defect. Kappa corrects for that.
Worked example: two inspectors, 50 parts
Fictional example. Inspectors A and B classify the same 50 parts (OK or NOK) without knowing each other's answers.
| B = OK | B = NOK | Total A | |
|---|---|---|---|
| A = OK | 20 | 5 | 25 |
| A = NOK | 3 | 22 | 25 |
| Total B | 23 | 27 | 50 |
- Observed agreement: po = (20 + 22) ÷ 50 = 42 ÷ 50 = 0.84.
- Chance agreement: pe = (25 × 23 + 25 × 27) ÷ 50² = (575 + 675) ÷ 2,500 = 0.50.
- Kappa = (0.84 − 0.50) ÷ (1 − 0.50) = 0.34 ÷ 0.50 = 0.68.
The two agree on 84% of the parts, but half of that would be expected by chance. A kappa of 0.68 shows only moderate to good agreement.
How to read the kappa value
| Kappa | Reading (usual MSA manual guideline) |
|---|---|
| above 0.75 | Good agreement |
| 0.40 to 0.75 | Marginal: acceptable depending on risk, improve |
| below 0.40 | Poor: the inspection system is not reliable |
In the example, 0.68 is in the marginal band. Before accepting, look at the 8 disagreeing parts (5 + 3): which parts, which defect, and whether the visual criterion was clear.
Comparing each inspector with the standard
When a reference classification exists (parts judged by an expert or by measurement), three rates are computed per inspector. Fictional example: inspector A assessed 25 good and 25 bad parts according to the standard.
| Standard = good | Standard = bad | |
|---|---|---|
| A accepts | 23 | 4 |
| A rejects | 2 | 21 |
- Effectiveness: (23 + 21) ÷ 50 = 88% correct decisions.
- Miss rate (escape): 4 ÷ 25 = 16% of bad parts were accepted. This is the error that reaches the customer.
- False alarm rate: 2 ÷ 25 = 8% of good parts were rejected. This error creates scrap and rework.
- Kappa against the standard: po = 0.88; pe = (27 × 25 + 23 × 25) ÷ 2,500 = 0.50; Kappa = 0.38 ÷ 0.50 = 0.76.
Usual MSA manual criteria: effectiveness above 90% acceptable and 80% to 90% marginal; miss rate up to 2% acceptable and up to 5% marginal; false alarm up to 5% acceptable and up to 10% marginal. Inspector A, with 88% effectiveness and a 16% miss rate, needs action despite a kappa of 0.76: kappa alone hides that the costliest error is the bad part accepted.
How to set up the study
- Choose 30 to 50 parts representing the process, about half good and half bad, including borderline parts and the line's real defects.
- Define the standard: classify each part by expert or measurement, and record the answer where inspectors cannot see it.
- Use 2 to 3 appraisers who really do the inspection, under normal lighting and time.
- Repeat 2 or 3 times with parts in random order and with a gap, to assess repeatability.
- Record on a check sheet, and compute kappa between appraisers, each against themselves, and against the standard. For more than two appraisers use Fleiss' kappa; for ordered ratings (e.g. 1 to 5), Kendall's coefficient. Software such as Minitab does the calculation.
- Act on the disagreements: define visual limit samples, improve lighting and defect descriptions, train with TWI and update the standard work.
Study size follows the same logic as sample size: few parts give an unstable kappa.
Where to apply it
- Visual inspection of finish, weld, paint and stitching.
- Go/no-go gauges and limit gauges.
- 100% sorting and final inspection before shipping.
- Customer requirements in APQP and PPAP, for measurement systems listed in the control plan.
If visual inspection is fragile, consider removing the human decision with poka-yoke or automated inspection. Go/no-go gauges remain under calibration. Lot sampling comes later, in the AQL plan: it does not fix an inspector who gets it wrong.
Common mistakes
- Using only percent agreement, without removing chance.
- A study with few parts, or only clearly good and clearly bad ones: kappa comes out inflated.
- Very unbalanced proportions of good and bad: kappa can be low even with high agreement.
- Appraisers who know they are being assessed and spend more time than normal.
- A doubtful reference standard: if the expert errs, the study measures the expert's error.
- Stopping at the number: without treating the disagreeing parts and opening corrective action, the study improves nothing.
Frequently asked questions
What is kappa in attribute agreement analysis?
An index that measures agreement between classifications after removing the agreement expected by chance. It ranges from −1 to 1.
What is the kappa formula?
Kappa = (po − pe) ÷ (1 − pe), where po is observed agreement and pe is agreement expected by chance.
What kappa is acceptable?
By the usual MSA manual guideline, above 0.75 is good agreement, 0.40 to 0.75 is marginal and below 0.40 is poor.
How many parts should the study use?
30 to 50 parts, about half good and half bad, including borderline parts, assessed by 2 to 3 appraisers in 2 or 3 trials.
What is the difference between kappa and Gauge R&R?
Kappa evaluates decisions (OK/NOK); Gauge R&R evaluates continuous measurements. Both belong to MSA.
Why does the miss rate matter more than kappa?
Because it shows how many bad parts pass and reach the customer. An appraiser can have a fair kappa and still let many defects through.
Sources
- AIAG. Measurement Systems Analysis (MSA), Reference Manual, 4th ed. Automotive Industry Action Group, 2010.
- COHEN, J. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 1960.
- MONTGOMERY, D. C. Introduction to Statistical Quality Control. Wiley.
Keep studying: trusted sources and links
Save this address: Manufatura Enxuta is your school for lean manufacturing. Come back whenever you need to study, with a complete article, a visual map and trusted sources. Follow one of the learning paths or check the A–Z glossary.
Want the tools ready to use?
Download the free "Problem Solving Kit" e-book, with PDCA, A3, 5 Whys and Ishikawa.
Download the e-book
About the author
Lean Manufacturing & Behavioral Management Specialist
Over 20 years in the automotive and metalworking industries (GM and Dana), Lean Manufacturing practitioner since 2006. SENAI instructor and mentor in Brazil’s Brasil Mais Produtivo program, delivering consulting, training and audits for 50+ companies, combining quality, productivity and people development.
