Calculate Cohen or Fleiss kappa with percent agreement, chance agreement, marginal diagnostics, interpretation, pasted counts, and clear statistic applicability.

Nominal ratings only: binary Cohen kappa or balanced Fleiss kappa. This tool does not calculate weighted kappa, ICC, confidence intervals or significance tests.

What Is Inter-Rater Reliability?

Inter-rater reliability (IRR) quantifies the degree to which independent raters, judges, or coders produce consistent assessments when evaluating the same set of subjects or items. It is one of the foundational quality checks in any study that relies on human judgment, from clinical diagnosis to content analysis to machine learning annotation pipelines. A high IRR value signals that the measurement instrument and rating criteria are well-defined enough that different observers reach the same conclusions independently.

IRR is distinct from intra-rater reliability, which measures the consistency of a single rater over time. Both fall under the broader umbrella of measurement reliability, but inter-rater reliability specifically addresses whether a construct can be rated consistently across people, not just within one person's repeated attempts.

Percent Agreement Formula

The simplest measure of IRR is percent agreement, which the calculator above computes directly. The formula is:

For two raters: Percent Agreement = (Items with matching classifications / Paired items) x 100. For balanced Fleiss counts, observed agreement is the mean proportion of agreeing rater pairs within subjects, not the percentage of subjects with unanimous ratings.

Percent agreement does not correct for a chance-agreement model. Independent uniform guessing in two categories gives 50% expected agreement, but unequal category probabilities change that expectation. Cohen kappa uses each rater's observed marginal distribution; Fleiss kappa uses pooled category proportions. Report observed agreement with the named coefficient rather than treating a chance model as proof of genuine consensus.

Cohen's Kappa

Jacob Cohen introduced this statistic in 1960 specifically to address the chance-agreement problem in percent agreement. Cohen's Kappa is designed for exactly two raters classifying items into mutually exclusive nominal categories. The formula is:

k = (P_o - P_e) / (1 - P_e)

Where P_o is the observed proportion of agreement and P_e is the expected proportion of agreement under chance. P_e is calculated from the marginal totals of the raters' classification matrix. Kappa values range from -1 to 1: a value of 1 indicates perfect agreement, 0 indicates agreement no better than chance, and negative values indicate systematic disagreement between the raters.

For ordinal or interval-scaled categories, Weighted Kappa extends the basic formula by assigning partial credit for near-misses. Linear weights penalize disagreements proportionally to their distance apart, while quadratic weights penalize proportionally to the square of the distance. Quadratic-weighted kappa is mathematically equivalent to the intraclass correlation coefficient under certain conditions, which makes it a useful bridge between categorical and continuous agreement measures.

Fleiss' Kappa

Published by Joseph Fleiss in 1971, Fleiss' Kappa generalizes the chance-corrected agreement framework to any fixed number of raters assigning nominal categories. Unlike Cohen's Kappa, which compares two specific raters cell by cell, Fleiss' Kappa works by computing the proportion of all rater pairs that agree for each item, then averaging across items and comparing to chance expectation. It does not require that the same individual raters evaluate every item, only that the same number of raters score each item. This makes it practical for studies where raters rotate across subjects, which is common in large-scale annotation projects.

Intraclass Correlation Coefficient (ICC)

When ratings are continuous or measured on an interval/ratio scale (e.g., pain scores from 0 to 10, or time measurements), kappa-based measures are inappropriate. The intraclass correlation coefficient (ICC) fills this role. ICC partitions the total variance in ratings into variance due to the subjects being rated and variance due to differences among raters. A high ICC means most of the variability comes from genuine differences between subjects rather than from rater inconsistency.

McGraw and Wong (1996) defined 10 forms of ICC based on three factors: the model (one-way random, two-way random, or two-way mixed effects), the type (single measures or average measures), and the definition (consistency or absolute agreement). The most commonly reported forms in research are ICC(2,1) for single-measure absolute agreement under a two-way random model, and ICC(3,1) for single-measure consistency under a two-way mixed model. Choosing the wrong form can substantially alter the reported reliability, so the specific ICC form should always be stated explicitly in any publication.

Krippendorff's Alpha

Developed by Klaus Krippendorff, this coefficient is the most flexible IRR measure available. It handles any number of raters, works across nominal, ordinal, interval, and ratio data, tolerates missing data (raters do not need to score every item), and applies a small-sample correction. When there are no missing values and the data are nominal, Krippendorff's Alpha produces values nearly identical to Fleiss' Kappa. Its main advantage is generality: a single metric can be applied consistently regardless of the study design, eliminating the need to switch between different statistics for different data types.

Choosing the Right IRR Measure

The correct statistic depends on the rating design, measurement scale, missingness and intended interpretation. This calculator implements only binary unweighted Cohen kappa and balanced Fleiss kappa; it does not calculate weighted kappa, ICC or Krippendorff alpha.

For two fixed raters and nominal categories, Cohen kappa is one option. Ordinal ratings may require justified linear or quadratic weights; continuous measurements may call for a specified ICC model. Multiple raters, missing ratings or different measurement scales require a method appropriate to that design. Krippendorff alpha needs an explicit distance function and does not automatically justify mixing measurement levels. Percent agreement remains useful alongside design-appropriate reliability measures.

Interpreting Kappa Values

The most cited benchmark is the Landis and Koch (1977) scale: below 0.00 is poor agreement, 0.00 to 0.20 is slight, 0.21 to 0.40 is fair, 0.41 to 0.60 is moderate, 0.61 to 0.80 is substantial, and 0.81 to 1.00 is almost perfect. An alternative scale by Cicchetti (1994) classifies below 0.40 as poor, 0.40 to 0.59 as fair, 0.60 to 0.74 as good, and 0.75 to 1.00 as excellent.

Both scales are based on expert opinion rather than statistical derivation, and neither accounts for the number of categories, number of raters, or prevalence distribution. A kappa of 0.65 in a two-category scheme is not directly comparable to a kappa of 0.65 in a ten-category scheme. These benchmarks are useful as rough guidelines, but any serious reliability report should present the full confusion matrix, marginal distributions, and category-specific agreement rates alongside the summary kappa value.

The Kappa Paradox

Cohen's Kappa has a well-documented counterintuitive behavior known as the kappa paradox, first described by Feinstein and Cicchetti in 1990. There are two forms. In the first, two raters can achieve very high percent agreement (e.g., 95%) yet produce a very low or even negative kappa. This happens when one category dominates the distribution (high prevalence), inflating the expected chance agreement so much that the observed agreement barely exceeds it. In the second form, two datasets with identical percent agreement can yield very different kappa values depending on how balanced the marginal distributions are.

The paradox is especially relevant in medical diagnostics and rare-event coding, where one category (e.g., "disease absent") vastly outnumbers the other. In these situations, prevalence-adjusted and bias-adjusted kappa (PABAK) or Gwet's AC1 may provide more stable estimates. Researchers should report prevalence indices and bias indices alongside kappa to help readers evaluate whether the paradox is distorting results.

Typical IRR Values by Field

Published coefficients vary with the task, category prevalence, rating design and uncertainty. Use results from closely matched studies for context; a generic numerical range for a discipline is not a reliability standard.

For annotation or educational scoring, choose an acceptance rule before reviewing the results and justify it for the task. State the coefficient, rating design, distribution and uncertainty rather than assuming one universal threshold.

Improving Low Inter-Rater Reliability

Low IRR scores almost always trace back to ambiguous rating criteria, insufficient rater training, or a mismatch between the construct and the measurement scale. The most effective intervention is calibration training, where raters independently score a shared set of anchor items, discuss disagreements, and refine the coding rules until agreement reaches an acceptable threshold before the main study begins. Providing detailed scoring rubrics with concrete examples of borderline cases reduces subjective interpretation. Reducing the number of categories (when scientifically justified) also increases agreement, since finer-grained scales introduce more opportunities for disagreement. Finally, measuring IRR at multiple time points during data collection can catch rater drift, where a rater's interpretation of the criteria gradually shifts over time.