Comparing the Wilcoxon-Mann-Whitney to alternative statistical tests

2022-04-26

In this document, we show that the Wilcoxon-Mann-Whitney test is comparable or superior to alternative methods.

Two alternative methods could be compared with the Wilcoxon-Mann-Whitney (WMW) test proposed by BioQC: the Kolmogorov-Smirnov (KS) test, and the Student’s t-test, or more particularly, the Welch’s test which does not assume equal sample number or equal variance, which is appropriate in the setting of gene expression studies.

  1. It is documented in statistics literature that the WMW test offers a higher power than the Kolmogorov-Smirnov test1,2.
  2. Compared with parameterized test methods such as the t-test, the WMW test is (a) resistance to monotone transformation, (b) suffers less from outliers, and (c) provides higher efficiency when many genes are profiled and the distribution of gene expression deviates from the normal distribution, which are important criteria in genome-wide expression data.

Based on these considerations, BioQC implements a computationally efficient version of the WMW test. In order not to confuse end-users, no alternative methods are implemented.

Nevertheless, in order to demonstrate the power of WMW test in comparison with the KS-test or the t-test, we performed the sensitivity benchmark described in the simulation studies, for the two alternative tests respectively.

**Figure 1:** Sensitivity benchmark. Expression levels of genes in the ovary signature are dedicately sampled randomly from normal distributions with different mean values. The lines show the enrichment score for the Wilcoxon-Mann-Whitney test, the t-test and the Kolmogorov-Smirnov test respectively. In the right panel, outliers were added by adding a random value to 1% of the simulated genes.

Figure 1: Sensitivity benchmark. Expression levels of genes in the ovary signature are dedicately sampled randomly from normal distributions with different mean values. The lines show the enrichment score for the Wilcoxon-Mann-Whitney test, the t-test and the Kolmogorov-Smirnov test respectively. In the right panel, outliers were added by adding a random value to 1% of the simulated genes.

As expected, the results suggest, that both the KS-test and the WMW-test are robust to noise, while the performance of the t-test drops significantly on noisy data. Additionally, the WMW-test appears to be superior to the KS-test for low expression differences.

Computational Performance

Since the KS-test is so slow, we did not replicate the sensitivity benchmark from the simulation studies using the enrichment score rank. While it takes BioQC about 4 seconds on a single thread to test all 155 signatures, it already takes the KS-test about 2 seconds to test a single signature.

##       test replications elapsed relative
## 2  runKS()            5  11.537    1.000
## 1 runWMW()            5  18.332    1.589

R Session Info

sessionInfo()
## R version 4.2.0 RC (2022-04-19 r82224)
## Platform: x86_64-pc-linux-gnu (64-bit)
## Running under: Ubuntu 20.04.4 LTS
## 
## Matrix products: default
## BLAS:   /home/biocbuild/bbs-3.15-bioc/R/lib/libRblas.so
## LAPACK: /home/biocbuild/bbs-3.15-bioc/R/lib/libRlapack.so
## 
## locale:
##  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
##  [3] LC_TIME=en_GB              LC_COLLATE=C              
##  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
##  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
##  [9] LC_ADDRESS=C               LC_TELEPHONE=C            
## [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
## 
## attached base packages:
## [1] stats4    stats     graphics  grDevices utils     datasets  methods  
## [8] base     
## 
## other attached packages:
##  [1] ggplot2_3.3.5         plyr_1.8.7            reshape2_1.4.4       
##  [4] hgu133plus2.db_3.13.0 rbenchmark_1.0.0      gplots_3.1.3         
##  [7] gridExtra_2.3         latticeExtra_0.6-29   lattice_0.20-45      
## [10] org.Hs.eg.db_3.15.0   AnnotationDbi_1.58.0  IRanges_2.30.0       
## [13] S4Vectors_0.34.0      testthat_3.1.4        limma_3.52.0         
## [16] RColorBrewer_1.1-3    BioQC_1.24.0          Biobase_2.56.0       
## [19] BiocGenerics_0.42.0   knitr_1.38           
## 
## loaded via a namespace (and not attached):
##  [1] httr_1.4.2             sass_0.4.1             pkgload_1.2.4         
##  [4] edgeR_3.38.0           bit64_4.0.5            jsonlite_1.8.0        
##  [7] gtools_3.9.2           bslib_0.3.1            brio_1.1.3            
## [10] assertthat_0.2.1       highr_0.9              blob_1.2.3            
## [13] GenomeInfoDbData_1.2.8 yaml_2.3.5             pillar_1.7.0          
## [16] RSQLite_2.2.12         glue_1.6.2             digest_0.6.29         
## [19] XVector_0.36.0         colorspace_2.0-3       htmltools_0.5.2       
## [22] pkgconfig_2.0.3        zlibbioc_1.42.0        purrr_0.3.4           
## [25] scales_1.2.0           jpeg_0.1-9             tibble_3.1.6          
## [28] KEGGREST_1.36.0        farver_2.1.0           generics_0.1.2        
## [31] ellipsis_0.3.2         cachem_1.0.6           withr_2.5.0           
## [34] cli_3.3.0              magrittr_2.0.3         crayon_1.5.1          
## [37] memoise_2.0.1          evaluate_0.15          fansi_1.0.3           
## [40] tools_4.2.0            lifecycle_1.0.1        stringr_1.4.0         
## [43] munsell_0.5.0          locfit_1.5-9.5         Biostrings_2.64.0     
## [46] compiler_4.2.0         jquerylib_0.1.4        GenomeInfoDb_1.32.0   
## [49] caTools_1.18.2         rlang_1.0.2            grid_4.2.0            
## [52] RCurl_1.98-1.6         labeling_0.4.2         bitops_1.0-7          
## [55] rmarkdown_2.14         gtable_0.3.0           DBI_1.1.2             
## [58] R6_2.5.1               dplyr_1.0.8            utf8_1.2.2            
## [61] fastmap_1.1.0          bit_4.0.4              rprojroot_2.0.3       
## [64] KernSmooth_2.23-20     desc_1.4.1             stringi_1.7.6         
## [67] Rcpp_1.0.8.3           vctrs_0.4.1            png_0.1-7             
## [70] tidyselect_1.1.2       xfun_0.30

References

  1. Irizarry, Rafael A., et al. “Gene set enrichment analysis made simple.”Statistical methods in medical research 18.6 (2009): 565-575.↩

  2. Filion, Guillaume J. “The signed Kolmogorov-Smirnov test: why it should not be used.” GigaScience 4.1 (2015): 1.↩