Possessing a manipulable representation of the world is a requirement for intelligent machines to plan, reason, and act in the world. Endowing computational systems, e.g., deep networks (DNs), with artificial intelligence (AI) capabilities is the goal of self-supervised learning (SSL). Immense optimism, fueled by early successes, funneled vast resources into SSL, which led to fast-paced but fragmented early developments.
As a result, today’s SSL research has become costly, slow-paced, and often akin to trial and error. To ensure that SSL research remains innovative and fail-proof, we present a unifying mathematical language that underpins SSL as a whole, and from which it is possible to explain and provably alleviate some of its limitations.
To that end, we bring forward a spectral graph theory of SSL that will rely on harmonic analysis and spectral graph theory. Under that view, SSL can be summarized as 1) estimating a relationship graph between the dataset samples and 2) learning to produce a smooth signal (from a DN) on that graph, as measured by the Dirichlet energy.
The vast collection of SSL methods fall under that formulation and differ only in how they estimate the graph, or how they measure the energy of the prediction on that graph. As part of this development, we also ground SSL to an ideal supervised learning task: SSL objectives minimize the expected risk over all possible downstream tasks. Finally, the practicality of our formulation is showcased by precisely pinpointing the benefits and pitfalls of each method, along with their underlying assumptions on the data and system design. For example, the impact of the data quality and sampling is studied, along with the role of various hyperparameters such as minibatch size, regularization, and data augmentation.
