Attribute Agreement Analysis Calculator

Use this calculator to evaluate categorical inspection ratings, such as pass/fail decisions or defect categories. Assess whether appraisers agree with themselves over repeated trials, with one another, and with an accepted reference classification. Results include percentage agreement, confidence intervals, and Fleiss' kappa.

How to calculate attribute agreement

1. Choose Attribute data. Enter numeric part identifiers in one column and each appraiser's ratings in a separate column.
2. Repeat each part identifier on a new row for each trial. Every appraiser must rate every part at least twice, with the same number of trials and complete ratings.
3. Select the identifier column under Part and the rating columns under Appraiser. This displays within-appraiser agreement.
4. If accepted classifications are available, select their column under Expert (Standard). Repeat the same reference classification on every row for a given part. In this calculator, selecting a standard also enables the between-appraiser and overall agreement tables.
5. Review agreement percentages and confidence intervals alongside kappa. Check disagreements with the standard before judging the inspection process.

Required data layout

Use one column per appraiser, rather than putting appraiser names in a single column. This small illustration shows two trials for each of two parts; a real study needs a representative selection of parts and categories.

Appraiser B changes the decision for part 1 between trials. Appraiser A is consistent on both parts and agrees with the standard. This illustrates why agreement within an appraiser and agreement with a standard answer different questions.

Try a complete attribute MSA example

The example contains 30 parts, each rated twice by Liam, Noor, and Sofia, plus a Standard column. Load the example, select Part, select Standard under Expert (Standard), and select all three appraisers. The ratings use categories 1 through 5; they are category labels, not continuous measurements. The reported kappa is unweighted, so it does not distinguish a one-category disagreement from a larger disagreement.

How to interpret agreement and kappa

• Within appraiser: does each person give the same rating to a part on repeated trials? Low agreement suggests inconsistent inspection decisions.
• Between appraisers: do different people classify the same parts consistently? Disagreements may indicate different interpretations of the criteria.
• Agreement with the standard: do the ratings match the accepted classification? Appraisers can agree with each other and still be wrong.
• Kappa: adjusts observed agreement for agreement expected from the category frequencies. A value of 1 represents perfect agreement; values near 0 indicate agreement near that expected by chance. Negative values indicate less agreement than expected by chance.

The assessment agreement tables count parts that meet the agreement condition across repeated ratings, rather than simply counting individual matching decisions. Inspect the 95% confidence intervals as well as the percentages: limited data can leave substantial uncertainty. Uneven category frequencies can affect kappa, so do not use one kappa cutoff as the sole acceptance criterion.

Designing a useful inspection study

Include representative parts and difficult cases near decision boundaries. Randomize inspection order and keep earlier ratings and reference classifications hidden from appraisers. Define the categories clearly, use consistent labels, and establish a trustworthy standard before assessing correctness. Without a standard, agreement measures consistency rather than accuracy.

For continuous measurements such as length or weight, use the Gage R&R calculator. To assess repeated numerical readings of a single reference part, use a Type 1 gage study.