Preconditioned spectral descent for deep learning

David E. Carlson, Edo Collins, Ya Ping Hsieh, Lawrence Carin, Volkan Cevher

Research output: Chapter in Book/Report/Conference proceedingConference contribution

14 Scopus citations

Abstract

Deep learning presents notorious computational challenges. These challenges include, but are not limited to, the non-convexity of learning objectives and estimating the quantities needed for optimization algorithms, such as gradients. While we do not address the non-convexity, we present an optimization solution that exploits the so far unused "geometry" in the objective function in order to best make use of the estimated gradients. Previous work attempted similar goals with preconditioned methods in the Euclidean space, such as L-BFGS, RMSprop, and ADAgrad. In stark contrast, our approach combines a non-Euclidean gradient method with preconditioning. We provide evidence that this combination more accurately captures the geometry of the objective function compared to prior work. We theoretically formalize our arguments and derive novel preconditioned non-Euclidean algorithms. The results are promising in both computational time and quality when applied to Restricted Boltzmann Machines, Feedforward Neural Nets, and Convolutional Neural Nets.
Original languageEnglish (US)
Title of host publicationAdvances in Neural Information Processing Systems
PublisherNeural information processing systems foundation
Pages2971-2979
Number of pages9
StatePublished - Jan 1 2015
Externally publishedYes

Fingerprint

Dive into the research topics of 'Preconditioned spectral descent for deep learning'. Together they form a unique fingerprint.

Cite this