Tralie, Christopher J.. Duke University(2017) "Geometric Multimedia Time Series"

Tralie, Christopher J.. Duke University(2017) "Geometric Multimedia Time Series", advisor: Saprio, Guillermo Harer, John

This thesis provides a new take on problems in multimedia times series analysis by using a shape-based perspective to quantify patterns in time, which is complementary to more traditional analysis-based time series techniques. Inspired by the dynamical systems community, the authors turn time series into shapes via sliding window embeddings, which the authors refer to as ``time-ordered point clouds'' (TOPCs). This framework has traditionally been used on a single 1D observation function for deterministic systems, but the authors generalize the sliding window technique so that it not only applies to multivariate data (e.g. videos), but that it also applies to data which is not stationary (e.g. music).

The geometry of their time-ordered point clouds can be quite informative. For periodic signals, the point clouds fill out topological loops, which, depending on harmonic content, reside on various high dimensional tori. For quasiperiodic signals, the point clouds are dense on a torus. The authors use modern tools from topological data analysis (TDA) to quantify degrees of periodicity and quasiperiodicity by looking at these shapes, and the authors show that this can be used to detect anomalies in videos of vibrating vocal folds. In the case of videos, this has the advantage of substantially reducing the amount of preprocessing, as no motion tracking is needed, and the technique operates on raw pixels. This is also one of the first known uses of persistent H2 in a high dimensional setting.

Periodic processes represent only a sliver of possible dynamics, and the authors also show that sequences of arbitrary normalized sliding window point clouds are approximately isometric between ''cover songs,'' or different versions of the same song, possibly with radically different spectral content. Surprisingly, in this application, an incredibly simple geometric descriptor based on self-similarity matrices performs the best, and it also enables us to use MFCC features for this task, which was previously thought not to be possible due to significant timbral differences that can exist between versions. When combined with traditional pitch-based features using similarity metric fusion, the authors obtain state of the art results on automatic cover song identification.

In addition to being used as a geometric descriptor, self-similarity matrices provide a unifying description of phenomena in time-ordered point clouds throughout their work, and the authors use them to illustrate properties such as recurrence, mirror symmetry in time, and harmonics in periodic processes. They also provide the base representation for designing isometry blind time warping algorithms, which the authors use to synchronize time-ordered point clouds that are shifted versions of each other in space without ever having to do a spatial alignment. In particular, the authors devise an algorithm that lower bounds the 1-stress between two time-ordered point clouds, which is related to the Gromov-Hausdorff distance.

Overall, the authors show a proof-of-concept and promise of the nascent field of geometric signal processing, which is worthy of further study in applications of music structure, multimodal data analysis, and video analysis.