Statistics – Machine Learning
Scientific paper
2011-11-18
Statistics
Machine Learning
Scientific paper
In this paper, we propose a second order optimization method to learn models where both the dimensionality of the parameter space and the number of training samples is high. In our method, we construct on each iteration a Krylov subspace formed by the gradient and an approximation to the Hessian matrix, and then use a subset of the training data samples to optimize over this subspace. As with the Hessian Free (HF) method of [7], the Hessian matrix is never explicitly constructed, and is computed using a subset of data. In practice, as in HF, we typically use a positive definite substitute for the Hessian matrix such as the Gauss-Newton matrix. We investigate the effectiveness of our proposed method on deep neural networks, and compare its performance to widely used methods such as stochastic gradient descent, conjugate gradient descent and L-BFGS, and also to HF. Our method leads to faster convergence than either L-BFGS or HF, and generally performs better than either of them in cross-validation accuracy. It is also simpler and more general than HF, as it does not require a positive semi-definite approximation of the Hessian matrix to work well nor the setting of a damping parameter. The chief drawback versus HF is the need for memory to store a basis for the Krylov subspace.
Povey Daniel
Vinyals Oriol
No associations
LandOfFree
Krylov Subspace Descent for Deep Learning does not yet have a rating. At this time, there are no reviews or comments for this scientific paper.
If you have personal experience with Krylov Subspace Descent for Deep Learning, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Krylov Subspace Descent for Deep Learning will most certainly appreciate the feedback.
Profile ID: LFWR-SCP-O-548429