Nonlinear Sciences – Adaptation and Self-Organizing Systems
Scientific paper
1996-01-08
Nonlinear Sciences
Adaptation and Self-Organizing Systems
35 pages, LaTeX, 2 PostScript figures
Scientific paper
I define a natural measure of the complexity of a parametric distribution relative to a given true distribution called the {\it razor} of a model family. The Minimum Description Length principle (MDL) and Bayesian inference are shown to give empirical approximations of the razor via an analysis that significantly extends existing results on the asymptotics of Bayesian model selection. I treat parametric families as manifolds embedded in the space of distributions and derive a canonical metric and a measure on the parameter manifold by appealing to the classical theory of hypothesis testing. I find that the Fisher information is the natural measure of distance, and give a novel justification for a choice of Jeffreys prior for Bayesian inference. The results of this paper suggest corrections to MDL that can be important for model selection with a small amount of data. These corrections are interpreted as natural measures of the simplicity of a model family. I show that in a certain sense the logarithm of the Bayesian posterior converges to the logarithm of the {\it razor} of a model family as defined here. Close connections with known results on density estimation and ``information geometry'' are discussed as they arise.
No associations
LandOfFree
A Geometric Formulation of Occam's Razor for Inference of Parametric Distributions does not yet have a rating. At this time, there are no reviews or comments for this scientific paper.
If you have personal experience with A Geometric Formulation of Occam's Razor for Inference of Parametric Distributions, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and A Geometric Formulation of Occam's Razor for Inference of Parametric Distributions will most certainly appreciate the feedback.
Profile ID: LFWR-SCP-O-713443