Enter the number of ratings in agreement, the total number of ratings, and the number of raters into the calculator to determine the inter-rater reliability.

Inter-Rater Reliability Formula

The following formula is used to calculate the inter-rater reliability between judges or raters.

IRR = TA / (TR*#R) *100

  • Where IRR is the inter-rater reliability (%)
  • TA is the total number of agreements in the ratings
  • TR is the total number of ratings given by each rater
  • #R is the number of raters

This formula should be used only in cases where there are more than 2 raters. When there are two raters, the formula simplifies to:

IRR = TA / (TR) *100

Inter-Rater Reliability Definition

Inter-rater reliability is defined as the ratio of the total number of agreements between raters and the total number of ratings. The term is used to describe how often a set of judges or raters agree on a certain score.

Example Problem

How to calculate inter-rater reliability?

First, determine the total number of ratings that were given by the raters. For this example, there were a total of 5 ratings given each by 3 judges, for one contestant.

Next, determine the total number of scores that were in agreement. In this case, the raters agreed on 8 total scores.

Finally, calculate the inter-rater reliability.

Using the formula above, the inter-rate reliability is calculated as:

IRR = TA / (TR*#R) *100

= 8/(3*5)*100

= 53.33 % agreement.