ICC

ICC: intra-class correlation

• https://en.wikipedia.org/wiki/Intraclass_correlation
  • Intraclass Correlation from Statistics How To
  • Shrout, P.E., Fleiss, J.L. (1979), Intraclass correlation: uses in assessing rater reliability, Psychological Bulletin, 86, 420-428.
  • ICC(1,1): each subject is measured by a different set of k randomly selected raters?;
  • ICC(2,1): k raters are randomly selected, then, each subject is measured by the same set of k raters;
• [math]\displaystyle{ Y_{ij} = \mu + \alpha_i + \varepsilon_{ij}, }[/math] where [math]\displaystyle{ \alpha_i }[/math] is the random effect from subject i,

[math]\displaystyle{ ICC(1,1) = \frac{\sigma_\alpha^2}{\sigma_\alpha^2+\sigma_\varepsilon^2}. }[/math]

• R packages (the main input is a matrix of n subjects x p raters. Each rater is a class/group)
  • psych: ICC()
  • irr: icc() for one-way or two-way model. This works on my data 30k by 58. The default option gives ICC(1). It can also compute ICC(A,1)/agreement and ICC(C,1)/consistency.
  • psy: icc(). No options are provided. I got an error: vector memory exhausted (limit reached?) when the data is 30k by 58.
  • rptR:

• Example: it shows ICC1 = ICC(1,1)

  R> library(psych)
  R> (o <- ICC(anxiety, lmer=FALSE) )
  Call: ICC(x = anxiety, lmer = FALSE)
  
  Intraclass correlation coefficients 
                           type  ICC   F df1 df2     p lower bound upper bound
  Single_raters_absolute   ICC1 0.18 1.6  19  40 0.094     -0.0405        0.44
  Single_random_raters     ICC2 0.20 1.8  19  38 0.056     -0.0045        0.45
  Single_fixed_raters      ICC3 0.22 1.8  19  38 0.056     -0.0073        0.48
  Average_raters_absolute ICC1k 0.39 1.6  19  40 0.094     -0.1323        0.70
  Average_random_raters   ICC2k 0.43 1.8  19  38 0.056     -0.0136        0.71
  Average_fixed_raters    ICC3k 0.45 1.8  19  38 0.056     -0.0222        0.73
  
   Number of subjects = 20     Number of Judges =  3
  
  R> library(irr)
  R> (o2 <- icc(anxiety, model="oneway")) # subjects be considered as random effects
   Single Score Intraclass Correlation
  
     Model: oneway 
     Type : consistency 
  
     Subjects = 20 
       Raters = 3 
       ICC(1) = 0.175
  
   F-Test, H0: r0 = 0 ; H1: r0 > 0 
     F(19,40) = 1.64 , p = 0.0939 
  
   95%-Confidence Interval for ICC Population Values:
    -0.077 < ICC < 0.484
  
  R> o$results["Single_raters_absolute", "ICC"]
  [1] 0.1750224
  R> o2$value
  [1] 0.1750224
  
  R> icc(anxiety, model="twoway", type = "consistency")
   Single Score Intraclass Correlation
  
     Model: twoway 
     Type : consistency 
  
     Subjects = 20 
       Raters = 3 
     ICC(C,1) = 0.216
  
   F-Test, H0: r0 = 0 ; H1: r0 > 0 
     F(19,38) = 1.83 , p = 0.0562 
  
   95%-Confidence Interval for ICC Population Values:
    -0.046 < ICC < 0.522
  R> icc(anxiety, model="twoway", type = "agreement")
   Single Score Intraclass Correlation
  
     Model: twoway 
     Type : agreement 
  
     Subjects = 20 
       Raters = 3 
     ICC(A,1) = 0.198
  
   F-Test, H0: r0 = 0 ; H1: r0 > 0 
   F(19,39.7) = 1.83 , p = 0.0543 
  
   95%-Confidence Interval for ICC Population Values:
    -0.039 < ICC < 0.494
  

  library(magrittr)
  library(ggplot2)
  
  set.seed(1)
  r1 <- round(rnorm(20, 10, 4))
  r2 <- round(r1 + 10 + rnorm(20, 0, 2))
  r3 <- round(r1 + 20 + rnorm(20, 0, 2))
  df <- data.frame(r1, r2, r3) %>% pivot_longer(cols=1:3)
  df %>% ggplot(aes(x=name, y=value)) + geom_point()
  
  df0 <- cbind(r1, r2, r3)
  icc(df0, model="oneway")  #  ICC(1) = -0.262  --> Negative. 
                            #  Shift can mess up the ICC. See wikipedia.
  icc(df0, model="twoway", type = "consistency")  # ICC(C,1) = 0.846 --> Make sense
  icc(df0, model="twoway", type = "agreement")    # ICC(A,1) = 0.106 --> Why?
  
  ICC(df0)
  Call: ICC(x = df0, lmer = T)
  
  Intraclass correlation coefficients 
                           type   ICC     F df1 df2       p lower bound upper bound
  Single_raters_absolute   ICC1 -0.26  0.38  19  40 9.9e-01     -0.3613      -0.085
  Single_random_raters     ICC2  0.11 17.43  19  38 2.9e-13      0.0020       0.293
  Single_fixed_raters      ICC3  0.85 17.43  19  38 2.9e-13      0.7353       0.920
  Average_raters_absolute ICC1k -1.65  0.38  19  40 9.9e-01     -3.9076      -0.307
  Average_random_raters   ICC2k  0.26 17.43  19  38 2.9e-13      0.0061       0.555
  Average_fixed_raters    ICC3k  0.94 17.43  19  38 2.9e-13      0.8929       0.972
  
   Number of subjects = 20     Number of Judges =  3
  

• Good ICC, bad CV or vice-versa, how to interpret?
• Intraclass Correlation Coefficient in R. icc() [irr package] and the function ICC() [psych package] are considered with a simple example.
• Introclass correlation (from Real Statistics Using Excel) with a simple example.
• Intraclass correlation coefficient vs. F-test (one-way ANOVA)?