Deep learning (DL) is a field of study within machine learning and signal processing that has been around for nearly 40 years. In the last 10 years, its progress on problems including speech-to-text, image recognition, image generation, and language generation has been phenomenal. This exponential progress has been driven by exciting engineering and algorithmic developments. At first, theory trying to identify what first principles could explain the extraordinary performance gains was lagging far behind the engineering improvements. More recently, the mathematics of DL has begun to catch up.
The first part of this special issue surveyed the interface between signal processing and DL, tracing how classical ideas find new meaning in the architecture and theory of modern neural networks. The response from the community, both in terms of submissions and readership, confirmed that this interface is an active and productive site of mathematical inquiry. This second part continues in the same spirit, but the articles collected here push further into territory where the mathematical foundations of DL remain incomplete, contested, or only beginning to crystallize.
More than half of the articles in this second part of our special issue focus on practical observations made during the last several years of great advances in DL, such as choices made within training, implicit bias in networks, and various architectural options. These articles investigate whether tools from signal processing can give us a mathematical understanding of the “rule-of-thumb” lessons of practical DL. Other articles in this issue focus on specific challenges that arise mainly due to the paradigm of deep networks as machine learning models, such as alignment and self-supervised learning (SSL). And finally, you will find many themes from both convex and nonconvex optimization throughout this issue.
The articles collected here push further into territory where the mathematical foundations of DL remain incomplete, contested, or only beginning to crystallize.
The first article, “Spectral Graph Theory: The Mathematics of Self-Supervised Learning,” by Balestriero and LeCun [A1], develops a unifying mathematical framework for SSL in which many SSL methods are interpreted as estimating a data graph and then learning representations that are smooth on that graph. It connects modern SSL objectives to Dirichlet energy minimization, ideal supervised targets, and classical spectral embedding methods, such as Laplacian eigenmaps and t-distributed stochastic neighbor embedding, helping explain both practical successes and limitations. The next three articles show different mathematical perspectives on training dynamics and implicit bias in deep neural networks. In “Training Neural Networks at Any Scale,” by Pethik et al. [A2], the authors examine the question of how to efficiently adjust learning hyperparameters across model scales, a crucial mechanism that has become instrumental in large-scale industrial applications. In the next article, “On the Convergence, Implicit Bias, and Edge of Stability of Gradient Descent in Deep Learning,” by Min et al. [A3], another “modern” optimization question is addressed: the analysis of gradient descent algorithms in the large step-size regime, where the local descent guarantees from classic optimization theory are replaced by oscillatory behavior and instability. Interestingly, in the class of nonconvex objectives defined by neural network training, such unstable regimes provide faster practical convergence. “An Overview of Low-Rank Structures in the Training and Adaptation of Large Models,” by Balzano et al. [A4], provides a tutorial introduction to research on low-rank structure in DL. The article distinguishes results on low-rank structure that emerge during training, such as low-rank activations due to masked training or “dropout,” versus structures that are enforced, such as low-rank adaptation for fine-tuning.
The next article “Aligning Large Language Models With Human Feedback: Mathematical Foundations and Algorithm Design,” by Li et al. [A5], provides an introduction to the mathematical foundations and algorithmic frameworks used to align large language models with human intentions, preferences, and values. It discusses standard alignment techniques, such as supervised fine-tuning, reinforcement learning with human feedback, and direct preference optimization, which have quickly become instrumental in today’s frontier models. A connection among sparse signal processing, convex programming, and DL is made in “Unveiling Hidden Convexity in Deep Learning: A Sparse Signal Processing Perspective,” by Zeger and Pilanci [A6]. The article shows that, for important classes of ReLU neural networks, seemingly non-convex training problems can be reformulated as equivalent convex optimization tasks closely related to Lasso and group Lasso models. This allows them to bring geometric interpretations to the solutions, analyzing polyhedral faces, hyperplanes, and polytopes relevant to the solution of the convex program. The relevant developments in nonsmooth optimization are discussed in “Computational Nonsmooth Analysis: Modern Applications and Recent Developments,” by Tian and So [A7]. The article focuses on mathematical and computational tools for analyzing optimization problems that are nonsmooth and nonconvex, which are central features of many modern learning systems. In the context of this special issue, it brings a contemporary optimization perspective to DL by emphasizing methods that go beyond the classical smooth theory traditionally used in signal processing and machine learning.
The final article of the issue provides a comprehensive overview of DL architectures for sequence modeling. “Sequence Modeling Architectures: Foundations,” by Afzal et al. [A8], focuses on core mathematical principles, reviewing the evolution of common sequence models, from recurrent neural networks to transformers and modern state-space models, as well as presenting general linear sequence mixers as a unifying framework to highlight the relationship among architectures.
We thank the authors of this second part of the special issue for their persistence through several rounds of reviews that each improved the quality and accessibility of their manuscripts. The reviewers of these manuscripts were especially involved, giving many important rounds of constructive feedback for which we are very grateful. We again thank Rebecca Wollman for her (truly) unending patience and logistical support. And finally, we thank Tülay Adali, editor-in-chief, and Selin Aviyente, area editor for special issues, for their dedication to our special issue on this important research topic at the intersection of signal processing and machine learning and for their commitment to ensuring that IEEE Signal Processing Magazine continues to deserve its excellent reputation as the best venue for tutorial-style signal processing material.
Appendix: Related Articles
- R. Balestriero and Y. LeCun, “Spectral graph theory: The mathematics of self-supervised learning,” IEEE Signal Process. Mag., vol. 43, no. 3, pp. 8–20, May 2026, doi: 10.1109/MSP.2026.3659059.
- T. Pethick, K. Antonakopoulos, T. S. Silveti-Falls, L. C. Vankadara, and V. Cevher, “Training neural networks at any scale: An exposition,” IEEE Signal Process. Mag., vol. 43, no. 3, pp. 21–36, May 2026, doi: 10.1109/MSP.2026.3667310.
- H. Min, L. E. MacDonald, and R. Vidal, “On the convergence, implicit bias, and edge of stability of gradient descent in deep learning: Reviewing recent progress,” IEEE Signal Process. Mag., vol. 43, no. 3, pp. 37–51, May 2026, doi: 10.1109/MSP.2026.3665010.
- L. Balzano et al., “An overview of low-rank structures in the training and adaptation of large models: From implicit low-dimensionality to efficient training and fine-tuning,” IEEE Signal Process. Mag., vol. 43, no. 3, pp. 52–70, May 2026, doi: 10.1109/MSP.2026.3666749.
- C. Li et al., “Aligning large language models with human feedback: Mathematical foundations and algorithm design,” IEEE Signal Process. Mag., vol. 43, no. 3, pp. 71–85, May 2026, doi: 10.1109/MSP.2026.3666824.
- E. Zeger and M. Pilanci, “Unveiling hidden convexity in deep learning: A sparse signal processing perspective,” IEEE Signal Process. Mag., vol. 43, no. 3, pp. 86–101, May 2026, doi: 10.1109/MSP.2026.3677783.
- L. Tian and A. M.-C. So, “Computational nonsmooth analysis: Modern applications and recent developments,” IEEE Signal Process. Mag., vol. 43, no. 3, pp. 102–114, May 2026, doi: 10.1109/MSP.2026.3662660.
- A. Afzal et al., “Sequence modeling architectures: Foundations,” IEEE Signal Process. Mag., vol. 43, no. 3, pp. 115–129, May 2026, doi: 10.1109/MSP.2026.3682808.
