Calculating Confidence Intervals and P-values for Various ICCs

Kilem L. Gwet, Ph.D.

2019-09-16

library(irrICC)
library(scales)

Abstract

irrICC is an R package that provides several functions for calculating various Intraclass Correlation Coefficients (ICC). This package follows closely the general framework of inter-rater and intra-rater reliability presented by Gwet (2014).

In this document, I like to show you how to obtain the confidence interval and the p-value associated with a particular Intraclass Correlation Coefficient (ICC) your previously obtained. To learn how to obtain the various ICCs implemented in this package, please refer to the User Guide.

Note that the package scales is used in this document for formating some numbers. But it not needed to run the irrICC package.

ICC under Model 1A

The following code gives us the inter-rater reliability coefficient of 0.252, the associated 95% confidence interval of (0.015,0.784). As for the p-value, the function pval.ICC1a() produces several p-values associated with various values of the null parameters rho.zero given by (0,0.1,0.3,0.5,0.7,0.9). Typically, researchers will calculate the p-value associated with 0 null value to test for statistical significance.
You can provide your own null values using the function pvals.ICC1a() whose default null value is 0. It’s why pvals.ICC1a(iccdata1) yields 0, 0.015.

  icc1a.fn(iccdata1)
#>      sig2s    sig2e     icc1a n r max.rep min.rep Mtot ov.mean
#> 1 1.761312 5.225529 0.2520899 5 4       3       1   40     5.2
  ci.ICC1a(iccdata1)
#>          lcb       ucb
#> 1 0.01495456 0.7841262
  pval.ICC1a(iccdata1)
#>   rho.zero       pval
#> 1      0.0 0.01533516
#> 2      0.1 0.13458658
#> 3      0.3 0.53041794
#> 4      0.5 0.81006323
#> 5      0.7 0.94659101
#> 6      0.9 0.99530941
  pvals.ICC1a(iccdata1)
#>   rho.zero       pval
#> 1        0 0.01533516

The 95% confidence level is what is offered by default. Can we obtain a 90% confidence interval instead? The answer is yes. This is achieved as follows:

  ci.ICC1a(iccdata1,conflev = 0.90)
#>          lcb       ucb
#> 1 0.04185384 0.7082716

Now, suppose you want to compute p-values based on Model 1A for the null values 0.15,0.25, and 0.45. You would proceed as follows:

  pvals.ICC1a(iccdata1,rho.zero = c(0.15,0.25,0.45))
#>   rho.zero      pval
#> 1     0.15 0.2298508
#> 2     0.25 0.4351265
#> 3     0.45 0.7556120

ICC under Model 1B

The following code gives us the intra-rater reliability coefficient of 0.562, the associated 95% confidence interval of (0,1). The function pval.ICC1b() gives you p-values for a predermined vector of null values (0.0,0.1,0.3,0.5,0.7,0.9). For example, the p-value associated with the null value 0.3 is given by 7.075e-02. If you want to supply your own null values you will need to use function pvals.ICC1b(), the default null value being 0.. Remember that Model 1B can only give you an intra-rater reliability coefficient. If you need an inter-rater reliability then you must use a different model.

  icc1b.fn(iccdata1)
#>     sig2r    sig2e     icc1b n r max.rep min.rep Mtot ov.mean
#> 1 4.32087 3.365846 0.5621217 5 4       3       1   40     5.2
  ci.ICC1b(iccdata1)
#>   lcb ucb
#> 1   0   1
  pval.ICC1b(iccdata1)
#>   rho.zero         pval
#> 1      0.0 4.686342e-06
#> 2      0.1 1.386945e-03
#> 3      0.3 7.074659e-02
#> 4      0.5 3.142024e-01
#> 5      0.7 6.484932e-01
#> 6      0.9 9.301346e-01
  pvals.ICC1b(iccdata1)
#>   gam.zero         pval
#> 1        0 4.686342e-06

Again, instead of the default 95% confidence interval, you may request a 90% confidence interval as follows:

  ci.ICC1b(iccdata1,conflev = 0.90)
#>   lcb ucb
#> 1   0   1

P-values associated with an arbitrary vector of null values (0.15,0.25,0.45) are calculated as follows:

  pvals.ICC1b(iccdata1,gam.zero = c(0.15,0.25,0.45))
#>   gam.zero        pval
#> 1     0.15 0.002962246
#> 2     0.25 0.020746475
#> 3     0.45 0.166885491

It follows that for the null value 0.25 you get p-value = 2.075e-02.

ICC under Model 2

Model 2 With Interaction

Under Model 2 with interaction, the confidence intervals and p-values are calculated as follows:

  icc2.inter.fn(iccdata1)
#>      sig2s    sig2r    sig2e    sig2sr    icc2r     icc2a n r max.rep
#> 1 2.018593 4.281361 1.315476 0.4067361 0.251627 0.8360198 5 4       3
#>   min.rep Mtot ov.mean
#> 1       1   40     5.2
  ci.ICC2r.inter(iccdata1)
#>          lcb       ucb
#> 1 0.02191927 0.7792666
  ci.ICC2a.inter(iccdata1)
#>         lcb       ucb
#> 1 0.5478536 0.9645078
  pval.ICC2r.inter(iccdata1)
#>   rho.zero         pval
#> 1      0.0 0.0009601902
#> 2      0.1 0.1191859085
#> 3      0.3 0.5526437466
#> 4      0.5 0.8194818940
#> 5      0.7 0.9480505063
#> 6      0.9 0.9953334341
  pvals.ICC2r.inter(iccdata1)
#>   rho.zero         pval
#> 1        0 0.0009601902
  pvals.ICC2a.inter(iccdata1)
#>   gam.zero         pval
#> 1        0 2.306507e-05
  pvals.ICC2r.inter(iccdata1,rho.zero = c(0.25,0.45))
#>   rho.zero      pval
#> 1     0.25 0.4589435
#> 2     0.45 0.7689111
  pvals.ICC2a.inter(iccdata1,gam.zero = c(0.25,0.45))
#>   gam.zero         pval
#> 1     0.25 0.0006793227
#> 2     0.45 0.0079884253

Model 2 Without Interaction

Under Model 2 without interaction, the confidence intervals and p-values are calculated as follows:

  icc2.nointer.fn(iccdata1)
#>      sig2s   sig2r    sig2e     icc2r    icc2a n r max.rep min.rep Mtot
#> 1 2.090769 4.34898 1.598313 0.2601086 0.801157 5 4       3       1   40
#>   ov.mean
#> 1     5.2
  ci.ICC2r.nointer(iccdata1)
#>          lcb       ucb
#> 1 0.02869092 0.7805637
  ci.ICC2a.nointer(iccdata1)
#>         lcb       ucb
#> 1 0.5505474 0.9639793
  pvals.ICC2r.nointer(iccdata1)
#>   rho.zero         pval
#> 1        0 5.413829e-06
  pvals.ICC2a.nointer(iccdata1)
#>   gam.zero         pval
#> 1        0 7.887974e-09

ICC under Model 3

Under Model 3 with interaction, the confidence intervals and p-values are calculated as follows:

  icc3.inter.fn(iccdata1)
#>      sig2s    sig2e    sig2sr     icc2r     icc2a n r max.rep min.rep Mtot
#> 1 2.257426 1.315476 0.2238717 0.5749097 0.6535279 5 4       3       1   40
#>   ov.mean
#> 1     5.2
  ci.ICC3r.inter(iccdata1)
#>         lcb       ucb
#> 1 0.2162702 0.9222461
  ci.ICC3a.inter(iccdata1)
#>        lcb       ucb
#> 1 0.142446 0.9091092
  pvals.ICC3r.inter(iccdata1)
#>   rho.zero         pval
#> 1        0 0.0009601902
  pvals.ICC3a.inter(iccdata1)
#>   gam.zero        pval
#> 1        0 0.008717435

Under Model 3 without interaction, the confidence intervals and p-values are calculated as follows:

  icc3.nointer.fn(iccdata1)
#>      sig2s    sig2e     icc2r     icc2a n r max.rep min.rep Mtot ov.mean
#> 1 2.241792 1.470638 0.6038611 0.6038611 5 4       3       1   40     5.2
  ci.ICC3r.nointer(iccdata1)
#>         lcb       ucb
#> 1 0.2494563 0.9248785
  pvals.ICC3r.nointer(iccdata1)
#>   rho.zero         pval
#> 1        0 5.413829e-06

References:

  1. Gwet, K.L. (2014, ISBN:978-0970806284). “Handbook of Inter-Rater Reliability,” 4th Edition. Advanced Analytics, LLC