The twenty ninth Conference on Neural Information Processing Systems (NIPS), a single track machine learning and computational neuroscience conference, was held in Montreal, during a relatively balmy winter week, spanning December 7th to 12th. The conference saw a record number of attendees. As illustrated in the graph, during the last 14 years, there has been a steady increase in the number of attendees in all three categories -- tutorials, conference, and workshops; and biggest growth this year was in tutorials where the attendance doubled compared to the previous year!
Figure 1: NIPS Growth
There were 1838 paper submissions, 1524 reviewers producing 10,625 reviews resulting in 403 accepted papers, an acceptance rate of about 22%. Apart from the two best papers and one best demo, the organizers honored 20 best reviewers, a tradition started since 2013 to maintain quality in the gargantuan review work.
Continuing the trend of recent years, the conference was overtaken by a disproportionate number of papers in deep learning, ranging from new types of models in this space to challenges encountered in novel applications. A few of examples include End to End Memory Network, Spatial Transformer Networks, and Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets.
The complexity of deep neural networks and the resources consumed in training them have increased rapidly in recent years, so it was no surprise that optimization was a big topic. A few examples include Deep Learning with Averaging Elastic SGD, <a href="Path SGD: Path Normalized Optimization in Deep Neural Networks, Probabilistic Line Searches for Stochastic Optimization, Training Very Deep Neural Networks, Efficient Exact Gradient Update for Training Deep Networks with Very Large Sparse Targets, Equilibrated Adaptive Learning Rate for Non-Convex Optimization, Learning Both Weights and Connections for Efficient Neural Networks, Incremental Methods for Additive Cost Function and papers at the Workshop on Non-Convex Optimization for Machine Learning: Theory and Practice.
A relatively new area that received considerable attention was probabilistic programming. This included an invited talk Probabilistic Machine Learning: Foundation and Frontiers by Zoubin Ghahramani. His paper in Nature provides a layman overview. In addition to the papers in the main conference, this topic was also covered in a tutorial -- Probabilistic Programming and in a workshop -- Black Box Learning and Inference.
Of course, as usual, there were a large number of fundamental contributions to machine learning including Learning theory and algorithms for forecasting non-stationary time series, Learning with Intelligent Teacher: Similarity Control and Knowledge Transfer, Randomized Block Krylov Methods for Stronger and Faster Approximate Singular Value Decomposition, Optimal Rates for Random Fourier Features, Lasso with Nonlinear Measurement is Equivalent to Lasso with Linear Measurement, Solving Random Quadratic System is as Easy as Linear, The Self-Normalized Estimation for Counterfactual Learning, and Principal Differences Analysis: Interpretable Characterization of Differences between Distributions.
Last but not least, the conference also included many interesting demos, the coolest of which clearly was Interactive Control of Diverse Complex Characters with Neural Networks, Here is the YouTube clip for your viewing pleasure. Enjoy!
Zak Shafran is the future editor-in-chief of the SLTC Newsletter. He is also a Research Scientist at Google.
