Stochastic spectral descent for restricted boltzmann machines

David Carlson, Volkan Cevher, Lawrence Carin

Research output: Chapter in Book/Report/Conference proceedingConference contribution

18 Scopus citations

Abstract

Restricted Boltzmann Machines (RBMs) are widely used as building blocks for deep learning models. Learning typically proceeds by using stochastic gradient descent, and the gradients are estimated with sampling methods. However, the gradient estimation is a computational bottleneck, so better use of the gradients will speed up the descent algorithm. To this end, we first derive upper bounds on the RBM cost function, then show that descent methods can have natural advantages by operating in the ℓ∞ and Shatten-∞ norm. We introduce a new method called "Stochastic Spectral Descent" that updates parameters in the normed space. Empirical results show dramatic improvements over stochastic gradient descent, and have only have a fractional increase on the per-iteration cost.
Original languageEnglish (US)
Title of host publicationJournal of Machine Learning Research
PublisherMicrotome Publishinginfo@mtome.com
Pages111-119
Number of pages9
StatePublished - Jan 1 2015
Externally publishedYes

Fingerprint Dive into the research topics of 'Stochastic spectral descent for restricted boltzmann machines'. Together they form a unique fingerprint.

Cite this