Empirical Squared Hellinger Distance Estimator and Generalizations to a Family of α-Divergence Estimators
<p>Empirical KLD estimator for two normal distributions.</p> "> Figure 2
<p>Failure of empirical PDF estimator.</p> "> Figure 3
<p>Empirical squared Hellinger estimator tests between 1D normal distributions.</p> "> Figure 4
<p>Empirical squared Hellinger estimator tests between 1D distributions.</p> "> Figure 5
<p>Empirical squared Hellinger estimator tests between 1D distributions.</p> "> Figure 6
<p>Empirical squared Hellinger estimator tests between 1D Cauchy and normal distributions.</p> "> Figure 7
<p>Comparisons between empirical and naive estimators for 1D normal distributions.</p> "> Figure 8
<p>Comparison of empirical <math display="inline"><semantics> <msub> <mi>D</mi> <mrow> <mi>K</mi> <mi>L</mi> </mrow> </msub> </semantics></math> estimator against empirical <math display="inline"><semantics> <msup> <mi>H</mi> <mn>2</mn> </msup> </semantics></math> estimator, <math display="inline"><semantics> <mrow> <mi>P</mi> <mo>=</mo> <mi>C</mi> <mi>a</mi> <mi>u</mi> <mi>c</mi> <mi>h</mi> <mi>y</mi> <mo>(</mo> <mn>0</mn> <mo>,</mo> <mn>1</mn> <mo>)</mo> <mo>,</mo> </mrow> </semantics></math><math display="inline"><semantics> <mrow> <mi>Q</mi> <mo>=</mo> <mi>N</mi> <mo>(</mo> <mn>0</mn> <mo>,</mo> <mn>1</mn> <mo>)</mo> </mrow> </semantics></math>.</p> "> Figure 9
<p>Vectorial squared Hellinger estimator (<math display="inline"><semantics> <mrow> <mi>k</mi> <mo>=</mo> <mn>5</mn> </mrow> </semantics></math>) tests on 2D normal distributions.</p> "> Figure 10
<p>Comparison of kNN-based squared Hellinger distance estimators.</p> "> Figure 11
<p>Two pairs of concentric Gaussians with invariant squared Hellinger distance.</p> "> Figure 12
<p><math display="inline"><semantics> <mi>α</mi> </semantics></math>-divergence estimator tests on 1D normal distributions.</p> "> Figure 13
<p>Comparisons between empirical and naive estimators for 1D normal distributions.</p> "> Figure 14
<p>Raw empirical TVD estimator.</p> ">
Abstract
:1. Introduction
2. Preliminaries on Divergences between Probability Distributions
3. Review of Empirical Sample-Based Kullback–Leibler Divergence Estimator of Continuous Distributions
4. Empirical Squared Hellinger Distance Estimator of Continuous Distributions
4.1. Estimator for 1D Data
4.2. Numerical Experiments
4.3. Estimator for Vectorial Data
4.4. Numerical Experiments for Vectorial Data
5. Empirical -Divergence Estimator of Continuous Distributions
5.1. Estimator for 1D Data
5.2. Numerical Experiments
5.3. Estimator for Vectorial Data
6. Limitation of the Proposed Methodologies and Uniqueness of the -Divergences
6.1. Failure of a Similar Estimator for Total Variation Distance
6.2. Uniqueness of -Divergences
7. Applications
7.1. Bounding the Neyman–Pearson Region by Hellinger Affinity
7.2. Estimating Eigenvalues of the Matrix Pencil for Inference in the Family of Concentric Gaussians
7.3. Stock Clustering and Visualization
7.4. Other Applications
8. Conclusions
Author Contributions
Funding
Conflicts of Interest
Appendix A. Shannon Entropy Estimator for 1D and Vectorial Data
Appendix B. Hellinger Affinity and Neyman–Pearson Region
Appendix C. Sufficient Information Eigenvalues for Inference between Concentric Gaussians
References
- Perez-Cruz, F. Kullback–Leibler divergence estimation of continuous distributions. In Proceedings of the IEEE International Symposium on Information Theory, Toronto, ON, Canada, 6–11 July 2008; pp. 1666–1670. [Google Scholar]
- Lee, Y.K.; Park, B.U. Estimation of Kullback–Leibler divergence by local likelihood. Ann. Inst. Stat. Math. 2006, 58, 327–340. [Google Scholar] [CrossRef]
- Anderson, N.; Hall, P.; Titterington, D. Two-sample test statistics for measuring discrepancies between two multivariate probability density functions using kernel-based density estimates. J. Multivar. Anal. 1994, 50, 41–54. [Google Scholar] [CrossRef] [Green Version]
- Nguyen, X.; Wainwright, M.J.; Jordan, M.I. Nonparametric estimation of the likelihood ratio and divergence functionals. In Proceedings of the IEEE International Symposium on Information Theory, Nice, France, 24–29 June 2007. [Google Scholar]
- Wang, Q.; Kulkarni, S.; Verdú, S. Divergence estimation of continuous distributions based on data-dependent partitions. IEEE Trans. Inf. Theory 2005, 51, 3064–3074. [Google Scholar] [CrossRef]
- Wang, Q.; Kulkarni, S.; Verdú, S. A nearest-neighbor approach to estimating divergence between continuous random vectors. In Proceedings of the IEEE International Symposium on Information Theory, Seattle, WA, USA, 9–14 July 2006; pp. 242–246. [Google Scholar]
- Yang, G.L.; Le Cam, L. Asymptotics in Statistics: Some Basic Concepts; Springer: Berlin, Germany, 2000. [Google Scholar]
- Kulllback, S.; Leibler, R.A. On information and sufficiency. Ann. Math. Stat. 1951, 22, 79–86. [Google Scholar] [CrossRef]
- Cichocki, A.; Amari, S. Families of alpha- beta- and gamma- divergences: Flexible and robust measures of similarities. Entropy 2010, 12, 1532–1568. [Google Scholar] [CrossRef] [Green Version]
- Binette, O. A note on reverse Pinsker inequalities. IEEE Trans. Inf. Theory 2019, 65, 4094–4096. [Google Scholar] [CrossRef]
- Sason, I.; Verdú, S. f-divergence inequalities. IEEE Trans. Inf. Theory 2016, 62, 5973–6006. [Google Scholar] [CrossRef]
- Gilardoni, G.L. On Pinsker’s type inequalities and Csiszar’s f-divergences, Part I: Second and Fourth-Order Inequalities. arXiv 2006, arXiv:cs/0603097. [Google Scholar]
- Poczos, B.; Schneider, J. On the Estimation of α-Divergences. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, Fort Lauderdale, FL, USA, 11–13 April 2011; Volume 15, pp. 609–617. [Google Scholar]
- Amari, S. α-Divergence is unique, belonging to both f-divergence and Bregman divergence classes. IEEE Trans. Inf. Theory 2009, 55, 4925–4931. [Google Scholar] [CrossRef]
- Gilberg, A.; Jain, L. If it ain’t broke, don’t fix it: Sparse metric repair. In Proceedings of the 55th Annual Allerton Conference on Communication, Control, and Computing (Allerton), Monticello, IL, USA, 3–6 October 2017; pp. 612–619. [Google Scholar]
- Ding, R. Visualizing Structures in Financial Time-Series Datasets through Affinity-Based Diffusion Transition Embedding. J. Financ. Data Sci. 2023, 5, 111–131. [Google Scholar] [CrossRef]
- Leonenko, N.N.; Pronzato, L.; Savani, V. A class of Rényi information estimators for multidimensional densities. Ann. Stat. 2008, 36, 2153–2182. [Google Scholar] [CrossRef]
- Lehmann, E.L.; Romano, J.P. Testing Statistical Hypotheses; Springer: New York, NY, USA, 2005. [Google Scholar]
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Ding, R.; Mullhaupt, A. Empirical Squared Hellinger Distance Estimator and Generalizations to a Family of α-Divergence Estimators. Entropy 2023, 25, 612. https://doi.org/10.3390/e25040612
Ding R, Mullhaupt A. Empirical Squared Hellinger Distance Estimator and Generalizations to a Family of α-Divergence Estimators. Entropy. 2023; 25(4):612. https://doi.org/10.3390/e25040612
Chicago/Turabian StyleDing, Rui, and Andrew Mullhaupt. 2023. "Empirical Squared Hellinger Distance Estimator and Generalizations to a Family of α-Divergence Estimators" Entropy 25, no. 4: 612. https://doi.org/10.3390/e25040612