Abstract
The overall Type I error computed based on the traditional means may be inflated if many hypotheses are compared simultaneously. The family-wise error rate (FWER) and false discovery rate (FDR) are some of commonly used error rates to measure Type I error under the multiple hypothesis setting. Many controlling FWER and FDR procedures have been proposed and have the ability to control the desired FWER/FDR under certain scenarios. Nevertheless, these controlling procedures become too conservative when only some hypotheses are from the null. Benjamini and Hochberg (J. Educ. Behav. Stat. 25:60–83, 2000) proposed an adaptive FDR-controlling procedure that adapts the information of the number of true null hypotheses (m 0) to overcome this problem. Since m 0 is unknown, estimators of m 0 are needed. Benjamini and Hochberg (J. Educ. Behav. Stat. 25:60–83, 2000) suggested a graphical approach to construct an estimator of m 0, which is shown to overestimate m 0 (see Hwang in J. Stat. Comput. Simul. 81:207–220, 2011). Following a similar construction, this paper proposes new estimators of m 0. Monte Carlo simulations are used to evaluate accuracy and precision of new estimators and the feasibility of these new adaptive procedures is evaluated under various simulation settings.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Benjamini, Y., Hochberg, Y.: Controlling the false discovery rate: a practical and powerful approach to multiple testing. J. R. Stat. Soc., Ser. B 57, 289–300 (1995)
Benjamini, Y., Hochberg, Y.: On the adaptive control of the false discovery rate in multiple testing with independent statistics. J. Educ. Behav. Stat. 25, 60–83 (2000)
Benjamini, Y., Liu, W.: A step-down multiple hypotheses testing procedure that controls the false discovery rate under independence. J. Stat. Plan. Inference 82, 163–170 (1999a)
Benjamini, Y., Liu, W.: A distribution-free multiple test procedure that controls the false discovery rate. Unpublished manuscript (1999b)
Benjamini, Y., Yekutieli, D.: The control of the false discovery rate in multiple testing under dependency. Ann. Stat. 29, 1165–1188 (2001)
Benjamini, Y., Krieger, A.M., Yekutieli, A.: Adaptive linear step-up procedures that control the false discovery rate. Biometrika 93(3), 491–507 (2006)
Gavrilov, Y., Benjamini, Y., Sarkar, S.K.: An adaptive step-down procedure with proven FDR control under independence. Ann. Stat. 37, 619–629 (2009)
Ge, Y., Dudoit, S., Speed, T.P.: Resampling-based multiple testing for microarray data hypothesis. Test 12, 1–44 (2003)
Genovese, C., Wasserman, L.: A stochastic process approach to false discovery control. Ann. Stat. 32, 1035–1061 (2004)
Holm, S.: A simple sequentially rejective multiple test procedure. Scand. J. Stat. 6, 65–70 (1979)
Hochberg, Y.: A sharper Bonferroni procedure for multiple tests of significance. Biometrika 75, 800–803 (1988)
Hochberg, Y., Benjamini, Y.: More powerful procedures for multiple significance testing. Stat. Med. 9, 811–818 (1990)
Hsueh, H.M., Chen, J.J., Kodell, R.L.: Comparison of methods for estimating the number of true null hypotheses in multiplicity testing. J. Biopharm. Stat. 13, 675–689 (2003)
Hsueh, H.M., Tsai, C.A., Chen, J.J.: Incorporating the number of true null hypotheses to improve power in multiple testing: application to gene microarray data. J. Stat. Comput. Simul. 77, 757–767 (2007)
Hwang, Y.T.: Estimating the number of true null hypotheses in multiple hypotheses testing. J. Stat. Comput. Simul. 81, 207–220 (2011)
Hwang, Y.T., Lai, J.J., Ou, S.T.: Evaluations of FWER-controlling methods in multiple hypothesis testing. J. Appl. Stat. 37, 1681–1694 (2010)
Hwang, Y.T., Chu, S.K., Ou, S.T.: Evaluations of FDR-controlling methods in multiple hypothesis testing. Stat. Comput. 21, 569–583 (2011)
Langaas, M., Lindqvist, B.H., Ferkingstad, E.: Estimating the proportion of true null hypotheses, with application to DNA microarray data. J. R. Stat. Soc., Ser. B 67, 555–572 (2005)
Liu, F., Sarkar, S.K.: A new adaptive method to control the false discovery rate. In: Bhattacharjee, M., Dhar, S.K., Subramanian, S. (eds.) Recent Advances in Biostatistics: False Discovery Rates, Survival Analysis, and Related Topics. Series in Biostatistics, vol. 4, pp. 3–26. World Scientific, New Jersey (2011)
Lu, X., Perkins, D.L.: Resampling strategy to improve the estimation of number of null hypotheses in FDR control under strong correlation structure. BMC Bioinform. 8, 157–169 (2007)
Meinshausen, N.: False discovery control for multiple tests of association under general dependence. Scand. J. Stat. 33(2), 227–237 (2006)
Meinshausen, N., Rice, J.: Estimating the proportion of false null hypotheses among a large number of independently tested hypotheses. Ann. Stat. 34, 373–393 (2006)
Miller, C.J., Genovese, C., Nichol, R.C., Wasserman, L., Connolly, A., Reichart, D., Hopkins, A., Schneider, J., Moore, A.: Controlling the false-discovery rate in astrophysical data analysis. Astron. J. 122, 3492–3505 (2001)
Pawian, Y., Michiels, S., Koscielny, S., Gusnanto, A., Ploner, A.: False discovery rate, sensitivity and sample size for microarray studies. Bioinformatics 21, 3017–3024 (2005)
Pounds, S.B.: Estimation and control of multiple testing error rates for microarray studies. Brief. Bioinform. 7, 25–36 (2006)
R Development Core Team: R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna (2008). http://www.R-project.org. ISBN 3-900051-07-0
Schweder, T., Spjøtvoll, E.: Plots of p-values to evaluate many test simultaneously. Biometrika 69, 493–502 (1982)
Seeger, P.: A note on a method for the analysis of significance en masse. Technometrics 10, 583–593 (1968)
Storey, J.D.: A direct approach to false discovery rates. J. R. Stat. Soc., Ser. B 64, 479–498 (2002)
Storey, J.D., Taylor, J.E., Siegmund, D.: Strong control, conservative point estimation, and simultaneous conservative consistency of false discovery rates: a unified approach. J. R. Stat. Soc., Ser. B 66, 187–205 (2004)
Turkheimer, F.E., Smith, C.B., Schmidt, K.: Estimation of the number of true null hypotheses in multivariate analysis of neuroimaging data. NeuroImage 13, 920–930 (2001)
Wang, C.C., Lin, Y.H., Hwang, Y.T.: A robust estimation of the proportion of true null hypotheses based on a beta mixture model. J. Chin. Stat. Assoc. 49, 1–17 (2011)
Yekutieli, D., Benjamini, Y.: Resampling-based false discovery rate controlling multiple test procedures for correlated test statistics. J. Stat. Plan. Inference 82, 171–196 (1999)
Acknowledgements
The authors gratefully acknowledge the Associate Editors and the referees for their insightful comments, which enhanced greatly the presentation and methodology of this paper. This research is partially supported by Nation Science Council Grant # NSC 96-2118-M-305-001 and # NSC 99-2118-M-305-001.
Author information
Authors and Affiliations
Corresponding author
Electronic Supplementary Material
Below is the link to the electronic supplementary material.
Rights and permissions
About this article
Cite this article
Hwang, YT., Kuo, HC., Wang, CC. et al. Estimating the number of true null hypotheses in multiple hypothesis testing. Stat Comput 24, 399–416 (2014). https://doi.org/10.1007/s11222-013-9377-5
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11222-013-9377-5