Mathematics – Optimization and Control
Scientific paper
2011-06-28
Mathematics
Optimization and Control
22 pages, 10 figures
Scientific paper
Stochastic Gradient Descent (SGD) is a popular algorithm that can achieve state-of-the-art performance on a variety of machine learning tasks. Several researchers have recently proposed schemes to parallelize SGD, but all require performance-destroying memory locking and synchronization. This work aims to show using novel theoretical analysis, algorithms, and implementation that SGD can be implemented without any locking. We present an update scheme called HOGWILD! which allows processors access to shared memory with the possibility of overwriting each other's work. We show that when the associated optimization problem is sparse, meaning most gradient updates only modify small parts of the decision variable, then HOGWILD! achieves a nearly optimal rate of convergence. We demonstrate experimentally that HOGWILD! outperforms alternative schemes that use locking by an order of magnitude.
Niu Feng
Re Christopher
Recht Benjamin
Wright Stephen J.
No associations
LandOfFree
HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent does not yet have a rating. At this time, there are no reviews or comments for this scientific paper.
If you have personal experience with HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent will most certainly appreciate the feedback.
Profile ID: LFWR-SCP-O-41854