Weighted Cohen's Kappa

Weighted Cohen's kappa is a measure of agreement between two raters who classify the same subjects using ordered categories. The ratings therefore form two paired ordinal samples.

In the case of an unweighted Cohen's Kappa, the variable to be measured by the two raters is usually a nominal variable. With a nominal variable, the categories can be distinguished, but there is no ranking between them.

Cohen's kappa takes into account whether the two raters measured the same thing or not, but it does not take into account the degree of disagreement. What if you don't have a nominal variable but an ordinal variable?

If you have an ordinal variable, that is, a variable in which the characteristics can be ordered, then of course you want to take that order into account.

Let's say your categories are "dissatisfied", "neutral", and "satisfied". There is a smaller difference between dissatisfied and neutral than between dissatisfied and satisfied. If you want to take the size of the difference into account, you have to use the weighted Cohen's Kappa.

Therefore, Cohen's kappa is generally used for nominal categories, whereas weighted Cohen's kappa is useful for ordinal categories when the size of a disagreement matters.

Reliability and validity

It is important to note that the weighted Cohen's Kappa can only tell you how reliably both raters are measuring the same thing. It cannot tell you whether what the two raters are measuring is the right thing!

So if both raters are pretty much always measuring the same thing, you would have a very high weighted Cohen's Kappa. However, the weighted Cohen's Kappa does not tell you whether this measurement corresponds to reality, i.e. whether the raters are measuring the right thing! In the first case we are talking about reliability. In the second case we speak of validity.

Calculating weighted Cohen's Kappa

How is the weighted Cohen's kappa calculated? Let's say two doctors have rated how satisfied they are with the therapeutic success of their patients. The doctors can answer with dissatisfied, neutral and satisfied.

Now you want to know how much agreement there is between the two doctors. Since we have an ordinal variable with the rank order dissatisfied, neutral and satisfied, we determine the agreement with the weighted Cohen's kappa.

The first step is to create a table with the frequencies of each response. We plot one rater on each axis. Here we have our two raters, each of whom rated whether they were dissatisfied, neutral or satisfied with a person's success.

Let's say a total of 75 patients have been evaluated. Now let's count how often each combination occurs. Let's say 17 times both raters are dissatisfied, 8 times rater 1 is dissatisfied and rater 2 is neutral, 4 times rater 1 is dissatisfied and rater 2 is satisfied and so on and so forth. For the ratings on the diagonal, both raters agree.

The weighted Cohen's kappa can be calculated using the following formula:

Here, w contains disagreement weights, f_o contains the observed frequencies, and f_e contains the frequencies expected from the raters' marginal distributions. Instead of frequencies, the corresponding observed and expected proportions can be used.

If we calculate weighted Cohen's kappa using proportions rather than frequencies, we would simply divide each frequency by the number of patients, i.e. 75, and have the observed probabilities.

But we still need the weights and the expected frequencies. Let's start with the expected frequencies.

Calculate expected frequency

To calculate the expected frequency, we first calculate the sums of the rows and columns. So we simply add up all the rows and all the columns.

For example, in the first row we get a sum of 29 with 17 + 8 + 4. We now divide this by 75 of the total number of cases.

We can now calculate the expected probability for each cell by multiplying the row probability by the column probability. So for the first cell we get 0.35 times 0.39 which is 0.13, for the second cell we get 0.44 times 0.39 which is 0.17.

Now, if we multiply each probability by 75, we get the expected frequencies.

Calculate weighting matrix

For the unweighted version, this disagreement-weight matrix consists of zeros on the diagonal and ones everywhere else. If both raters gave the same answer, the weight is zero; otherwise it is one. Thus, every disagreement is treated equally, regardless of how far apart the ratings are.

The linear weighting matrix can be calculated using the following formula. Let i be the index for the rows and j for the columns. k is the number of categories, in our case 3.

So now scores that are close together are weighted less than scores that are far apart.

Linear and quadratic weighting

What about quadratic weighting? If we use quadratic weighting instead of linear weighting, the distances are simply squared again. In this way, scores that are far apart are weighted even more heavily in relation to scores that are close together than in the linear case. The weighting matrix is then obtained with the following matrix.

So we can now decide whether to use no weighting, linear weighting or quadratic weighting. Linear weights increase in direct proportion to the distance between categories, while quadratic weights penalize larger distances more strongly. The weighting rule should be chosen before examining the result because it can noticeably change kappa. Custom weights may also be used when category differences have a well-defined subject-matter meaning. We will continue with linear weighting.

Calculate weighted kappa

We can now calculate the weighted kappa. We have the weighting matrix, the observed frequency and the expected frequency. Let's start with the sum in the figure below. We simply multiply each cell of the weighting matrix by the corresponding cell of the observed frequency and add them up. So 0 times 17 + 0.5 times 8 to finally 0 times 9.

We now do the same with the weighting matrix and the expected frequency. 0 times 10.05 plus 0.5 times 12.76 and finally 0 times 3.84. If we now calculate everything, we get a weighted kappa of 0.396.

A value of 1 means perfect agreement, 0 means no more agreement than expected from the marginal distributions, and a negative value means less agreement than expected. Descriptive labels such as “moderate” are only rough guidelines and should not replace subject-specific criteria. Kappa can also be affected by how common each category is and by differences between the raters' marginal distributions. When available, report a confidence interval as well as the estimate.

Calculating Cohen's weighted kappa with numiqo

To calculate weighted Cohen's Kappa online, simply go to the Statistics Calculator, copy your own data into this table, and click on the Reliability tab.

numiqo automatically tries to assign the appropriate scale level to the data, in this case numiqo assumes that the data are nominal. If we clicked on Rater 1 and Rater 2, numiqo would calculate the unweighted normal Cohen's kappa. However, in our case these are ordinal variables. So we simply change the scale level to ordinal.

If we now click on both raters, the weighted Cohen's kappa is calculated. We can now choose whether we want linear or quadratic weighting. Here we see the cross table, which shows us how often each combination occurs. Then we get the result for weighted Cohen's kappa.

If you're not sure how to interpret the result, click on Summary in Words. For this separate dataset, the screenshot reports a linearly weighted Cohen's kappa of κ = 0.50 with a 95% confidence interval from 0.28 to 0.72. The manual example above gives κ = 0.396. Always state the weighting method because linear and quadratic weights generally produce different values.