Adaptive Momentum for Neural Network Optimization

Simple item page
dc.contributor.advisorAn, Aijun
dc.contributor.authorRashidi, Zana
dc.date.accessioned2020-05-11T12:56:16Z
dc.date.available2020-05-11T12:56:16Z
dc.date.copyright2019-12
dc.date.issued2020-05-11
dc.date.updated2020-05-11T12:56:16Z
dc.degree.disciplineComputer Science
dc.degree.levelMaster's
dc.degree.nameMSc - Master of Science
dc.description.abstractIn this thesis, we develop a novel and efficient algorithm for optimizing neural networks inspired by a recently proposed geodesic optimization algorithm. Our algorithm, which we call Stochastic Geodesic Optimization (SGeO), utilizes an adaptive coefficient on top of Polyaks Heavy Ball method effectively controlling the amount of weight put on the previous update to the parameters based on the change of direction in the optimization path. Experimental results on strongly convex functions with Lipschitz gradients and deep Autoencoder benchmarks show that SGeO reaches lower errors than established first-order methods and competes well with lower or similar errors to a recent second-order method called K-FAC (Kronecker-Factored Approximate Curvature). We also incorporate Nesterov style lookahead gradient into our algorithm (SGeO-N) and observe notable improvements. We believe that our research will open up new directions for high-dimensional neural network optimization where combining the efficiency of first-order methods and the effectiveness of second-order methods proves a promising avenue to explore.
dc.identifier.urihttps://hdl.handle.net/10315/37485
dc.languageen
dc.rightsAuthor owns copyright, except where explicitly noted. Please contact the author directly with licensing requests.
dc.subjectComputer science
dc.subject.keywordsMachine learning
dc.subject.keywordsOptimization
dc.subject.keywordsMomentum
dc.subject.keywordsNeural networks
dc.subject.keywordsGeodesics
dc.subject.keywordsArtificial intelligence
dc.titleAdaptive Momentum for Neural Network Optimization
dc.typeElectronic Thesis or Dissertation

Files

Original bundle
Now showing 1 - 1 of 1
Thumbnail Image
Name:
Rashidi_Zana_2019_MSc.pdf
Size:
2.98 MB
Format:
Adobe Portable Document Format
Download
License bundle
Now showing 1 - 2 of 2
No Thumbnail Available
Name:
license.txt
Size:
1.83 KB
Format:
Plain Text
Description:
Download
No Thumbnail Available
Name:
YorkU_ETDlicense.txt
Size:
3.36 KB
Format:
Plain Text
Description:
Download

Collections

Computer Science and Engineering

All items in the YorkSpace institutional repository are protected by copyright, with all rights reserved except where explicitly noted.