An Anomaly Detection Framework with Compressed Transformer Architecture for Tiny ML” presented by Dr. Luca Barbieri, based on the IEEE Xplore® article titled “A Tiny Transformer-Based Anomaly Detection Framework for IoT Solutions”. Published in the IEEE Journal of Selected Topics in Signal Processing, July 2024. Visit the article here.
Transformers have recently revolutionized the artificial intelligence landscape, providing state-of-the-art performances in several tasks, including anomaly detection. The latter feature is fundamental to monitoring the status of industrial systems and promptly raising an alarm whenever malfunctions are detected.
However, transformers often entail huge computational complexity and memory requirements that are hardly available in tiny, low-computation, low-energy, and battery-powered devices, calling for dedicated countermeasures to reduce energy consumption and memory footprints.
This webinar presents a compression framework tailored for transformer-based anomaly detection tools, making them viable to be run on energy-limited devices.
The presenter will describe how the anomaly transformer can be compressed using knowledge distillation, substantially reducing the number of training parameters of the model, and consequently its computational and memory complexity. Simulation and experimental results show the effectiveness of the developed compression framework, comparing its results against other baselines.
