Renyi Divergence and Kullback-Leibler Divergence

被引:906
作者
van Erven, Tim [1 ]
Harremoes, Peter [2 ]
机构
[1] Univ Paris 11, Dept Math, F-91405 Orsay, France
[2] Copenhagen Business Coll, DK-1358 Copenhagen, Denmark
关键词
alpha-divergence; Bhattacharyya distance; information divergence; Kullback-Leibler divergence; Pythagorean inequality; Renyi divergence; INFORMATION; ENTROPY; PROBABILITY; DISTRIBUTIONS; CONVERGENCE; STATISTICS; PINSKERS; RATES;
D O I
10.1109/TIT.2014.2320500
中图分类号
TP [自动化技术、计算机技术];
学科分类号
0812 ;
摘要
Renyi divergence is related to Renyi entropy much like Kullback-Leibler divergence is related to Shannon's entropy, and comes up in many settings. It was introduced by Renyi as a measure of information that satisfies almost the same axioms as Kullback-Leibler divergence, and depends on a parameter that is called its order. In particular, the Renyi divergence of order 1 equals the Kullback-Leibler divergence. We review and extend the most important properties of Renyi divergence and Kullback-Leibler divergence, including convexity, continuity, limits of sigma-algebras, and the relation of the special order 0 to the Gaussian dichotomy and contiguity. We also show how to generalize the Pythagorean inequality to orders different from 1, and we extend the known equivalence between channel capacity and minimax redundancy to continuous channel inputs (for all orders) and present several other minimax results.
引用
收藏
页码:3797 / 3820
页数:24
相关论文
共 60 条
[41]  
Renyi A., 1967, Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability, Volume 1: Statistics, V1, P531
[42]  
Rnyi A., 1961, P 4 BERK S MATH STAT, P547, DOI DOI 10.1021/JP106846B
[43]  
Rockafellar R. T., 1970, Convex Analysis
[44]  
Rostamizadeh A., 2009, P 25 C UNCERTAINTY A, P367
[45]   A SOURCE MATCHING APPROACH TO FINDING MINIMAX CODES - COMMENT [J].
RYABKO, BY .
IEEE TRANSACTIONS ON INFORMATION THEORY, 1981, 27 (06) :780-781
[46]  
Shayevitz O., 2010, ARXIV10124401V1
[47]  
Shayevitz O, 2011, IEEE INT SYMP INFO, P894, DOI 10.1109/ISIT.2011.6034266
[48]  
Shtar'kov Yu. M., 1987, Problems of Information Transmission, V23, P175
[49]   INFORMATION RADIUS [J].
SIBSON, R .
ZEITSCHRIFT FUR WAHRSCHEINLICHKEITSTHEORIE UND VERWANDTE GEBIETE, 1969, 14 (02) :149-&
[50]  
Sion M., 1958, PAC J MATH, V8, P171, DOI 10.2140/pjm.1958.8.171