Plenary Talk: Variational Splines and Stochastic Processes: From Kernel Methods (and Norbert Wiener) to ReLU Networks
This plenary talk by Michael Unser explores the mathematical bridge connecting variational splines, stochastic processes, kernel methods, and deep neural networks. The primary goal is to establish a unified variational framework that explains how classical Minimum Mean Square Error (MMSE) linear spline estimators, total variation regularization, and modern artificial intelligence architectures interrelate across high- and low-dimensional function spaces.
The presentation details how traditional kernel methods naturally align with Gaussian stochastic processes, whereas deep Rectified Linear Unit (ReLU) networks can be mathematically interpreted as multidimensional spline functions. By framing continuous-domain reconstruction problems through variational optimization, the talk provides theoretical evidence showing that neural network models essentially correspond to sparse stochastic processes. This perspective clarifies how regularization choices directly influence model representation and complexity across classical and learning-based signal processing.
Ultimately, the key takeaway is that unifying splines, kernel methods, and neural networks under a single variational umbrella bridges classical signal estimation with modern deep learning. This comprehensive mathematical framework equips signal processing researchers with deeper foundational insights to analyze, interpret, and optimize next-generation deep neural network architectures.
