Relative Entropy and Statistics

Details Relative Entropy and Statistics Relative Entropy and Statistics

2008-08-29

arxiv.org/abs/0808.4111v2

Bavaud F. (2009) Information Theory, Relative Entropy and Statistics. In: Sommaruga G. (editor): Formal Theories of Informatio

Computer Science

Information Theory

31 pages. 2 figures.

Scientific paper

Formalising the confrontation of opinions (models) to observations (data) is the task of Inferential Statistics. Information Theory provides us with a basic functional, the relative entropy (or Kullback-Leibler divergence), an asymmetrical measure of dissimilarity between the empirical and the theoretical distributions. The formal properties of the relative entropy turn out to be able to capture every aspect of Inferential Statistics, as illustrated here, for simplicity, on dices (= i.i.d. process with finitely many outcomes): refutability (strict or probabilistic): the asymmetry data / models; small deviations: rejecting a single hypothesis; competition between hypotheses and model selection; maximum likelihood: model inference and its limits; maximum entropy: reconstructing partially observed data; EM-algorithm; flow data and gravity modelling; determining the order of a Markov chain.

Affiliated with

Also associated with

No associations

Rating

Relative Entropy and Statistics does not yet have a rating. At this time, there are no reviews or comments for this scientific paper.

If you have personal experience with Relative Entropy and Statistics, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Relative Entropy and Statistics will most certainly appreciate the feedback.

Rate now

Comments { 0 }

Profile ID: LFWR-SCP-O-543493