A Bregman Extension of quasi-Newton updates I: An Information Geometrical framework

Details A Bregman Extension of quasi-Newton updates I: An Information Geometrical framework A Bregman Extension of quasi-Newton updates I: An Information Geometrical framework

2010-10-14

arxiv.org/abs/1010.2847v1

Statistics

Computation

34 pages, 4 figures

Scientific paper

We study quasi-Newton methods from the viewpoint of information geometry induced associated with Bregman divergences. Fletcher has studied a variational problem which derives the approximate Hessian update formula of the quasi-Newton methods. We point out that the variational problem is identical to optimization of the Kullback-Leibler divergence, which is a discrepancy measure between two probability distributions. The Kullback-Leibler divergence for the multinomial normal distribution corresponds to the objective function Fletcher has considered. We introduce the Bregman divergence as an extension of the Kullback-Leibler divergence, and derive extended quasi-Newton update formulae based on the variational problem with the Bregman divergence. As well as the Kullback-Leibler divergence, the Bregman divergence introduces the information geometrical structure on the set of positive definite matrices. From the geometrical viewpoint, we study the approximation Hessian update, the invariance property of the update formulae, and the sparse quasi-Newton methods. Especially, we point out that the sparse quasi-Newton method is closely related to statistical methods such as the EM-algorithm and the boosting algorithm. Information geometry is useful tool not only to better understand the quasi-Newton methods but also to design new update formulae.

Affiliated with

Also associated with

No associations

Rating

A Bregman Extension of quasi-Newton updates I: An Information Geometrical framework does not yet have a rating. At this time, there are no reviews or comments for this scientific paper.

If you have personal experience with A Bregman Extension of quasi-Newton updates I: An Information Geometrical framework, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and A Bregman Extension of quasi-Newton updates I: An Information Geometrical framework will most certainly appreciate the feedback.

Rate now

Comments { 0 }

Profile ID: LFWR-SCP-O-285166