Adaptive Stochastic Optimization: A Framework for Analyzing Stochastic Optimization Algorithms

Optimization lies at the heart of machine learning (ML) and signal processing (SP). Contemporary approaches based on the stochastic gradient (SG) method are nonadaptive in the sense that their implementation employs prescribed parameter values that need to be tuned for each application. This article summarizes recent research and motivates future work on adaptive stochastic optimization methods, which have the potential to offer significant computational savings when training largescale systems.

Optimization lies at the heart of machine learning (ML) and signal processing (SP). Contemporary approaches based on the stochastic gradient (SG) method are nonadaptive in the sense that their implementation employs prescribed parameter values that need to be tuned for each application. This article summarizes recent research and motivates future work on adaptive stochastic optimization methods, which have the potential to offer significant computational savings when training large-scale systems.

The success of stochastic optimization algorithms for solving problems arising in ML and SP are now widely recognized. Scores of articles have appeared in recent years as researchers aim to build on fundamental methodologies, such as the SG method [26]. The motivation and scope of many of these efforts have been captured in various books and review articles;