# ROC curve

• Binary case:
• Y = true positive rate = sensitivity,
• X = false positive rate = 1-specificity = 假陽性率
• Area under the curve AUC from the wikipedia: the probability that a classifier will rank a randomly chosen positive instance higher than a randomly chosen negative one (assuming 'positive' ranks higher than 'negative').
$\displaystyle{ A = \int_{\infty}^{-\infty} \mbox{TPR}(T) \mbox{FPR}'(T) \, dT = \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} I(T'\gt T)f_1(T') f_0(T) \, dT' \, dT = P(X_1 \gt X_0) }$

where $\displaystyle{ X_1 }$ is the score for a positive instance and $\displaystyle{ X_0 }$ is the score for a negative instance, and $\displaystyle{ f_0 }$ and $\displaystyle{ f_1 }$ are probability densities as defined in previous section.

# Survival data

'Survival Model Predictive Accuracy and ROC Curves' by Heagerty & Zheng 2005

• Recall Sensitivity= $\displaystyle{ P(\hat{p_i} \gt c | Y_i=1) }$, Specificity= $\displaystyle{ P(\hat{p}_i \le c | Y_i=0 }$), $\displaystyle{ Y_i }$ is binary outcomes, $\displaystyle{ \hat{p}_i }$ is a prediction, $\displaystyle{ c }$ is a criterion for classifying the prediction as positive ($\displaystyle{ \hat{p}_i \gt c }$) or negative ($\displaystyle{ \hat{p}_i \le c }$).
• For survival data, we need to use a fixed time/horizon (t) to classify the data as either a case or a control. Following Heagerty and Zheng's definition (Incident/dynamic), Sensitivity(c, t)= $\displaystyle{ P(M_i \gt c | T_i = t) }$, Specificity= $\displaystyle{ P(M_i \le c | T_i \gt 0 }$) where M is a marker value or $\displaystyle{ Z^T \beta }$. Here sensitivity measures the expected fraction of subjects with a marker greater than c among the subpopulation of individuals who die at time t, while specificity measures the fraction of subjects with a marker less than or equal to c among those who survive beyond time t.
• The AUC measures the probability that the marker value for a randomly selected case exceeds the marker value for a randomly selected control
• ROC curves are useful for comparing the discriminatory capacity of different potential biomarkers.

# Confusion matrix, Sensitivity/Specificity/Accuracy

 Predict 1 0 True 1 TP FN Sens=TP/(TP+FN)=Recall FNR=FN/(TP+FN) 0 FP TN Spec=TN/(FP+TN), 1-Spec=FPR PPV=TP/(TP+FP) FDR=FP/(TP+FP) NPV=TN/(FN+TN) N = TP + FP + FN + TN
• Sensitivity 敏感度 = TP / (TP + FN) = Recall
• Specificity 特異度 = TN / (TN + FP)
• Accuracy = (TP + TN) / N
• False discovery rate FDR = FP / (TP + FP)
• False negative rate FNR = FN / (TP + FN)
• False positive rate FPR = FP / (FP + TN)
• True positive rate = TP / (TP + FN)
• Positive predictive value (PPV) = TP / # positive calls = TP / (TP + FP) = 1 - FDR
• Negative predictive value (NPV) = TN / # negative calls = TN / (FN + TN)
• Prevalence 盛行率 = (TP + FN) / N.
• Note that PPV & NPV can also be computed from sensitivity, specificity, and prevalence:
$\displaystyle{ \text{PPV} = \frac{\text{sensitivity} \times \text{prevalence}}{\text{sensitivity} \times \text{prevalence}+(1-\text{specificity}) \times (1-\text{prevalence})} }$
$\displaystyle{ \text{NPV} = \frac{\text{specificity} \times (1-\text{prevalence})}{(1-\text{sensitivity}) \times \text{prevalence}+\text{specificity} \times (1-\text{prevalence})} }$

# genefilter package and rowpAUCs function

• rowpAUCs function in genefilter package. The aim is to find potential biomarkers whose expression level is able to distinguish between two groups.
# source("http://www.bioconductor.org/biocLite.R")
# biocLite("genefilter")
library(Biobase) # sample.ExpressionSet data
data(sample.ExpressionSet)

library(genefilter)
r2 = rowpAUCs(sample.ExpressionSet, "sex", p=0.1)
plot(r2) # first gene, asking specificity = .9

r2 = rowpAUCs(sample.ExpressionSet, "sex", p=1.0)
plot(r2) # it won't show pAUC

r2 = rowpAUCs(sample.ExpressionSet, "sex", p=.999)
plot(r2) # pAUC is very close to AUC now


# Comparison of two AUCs

• Statistical Assessments of AUC. This is using the pROC::roc.test function.
• prioritylasso. It is using roc(), auc(), roc.test(), plot.roc() from the pROC package. The calculation based on the training data is biased so we need to report the one based on test data.

# Confidence interval of AUC

How to get an AUC confidence interval. pROC package was used.

# AUC can be a misleading measure of performance

AUC is high but precision is low (i.e. FDR is high). https://twitter.com/michaelhoffman/status/1398380674206285830?s=09.

# Class comparison problem

• compcodeR: RNAseq data simulation, differential expression analysis and performance comparison of differential expression methods
• Polyester: simulating RNA-seq datasets with differential transcript expression, github, HTML