Learning When to Speak: Human-Inspired Unit Sensing for Simultaneous Speech Translation

Recent simultaneous speech translation (SimulST) systems have demonstrated remarkable success in generating high-quality translations with the aid of large language models (LLMs). However, most existing works rely on heuristic-based policies to decide whether to write translations or read more source input, resulting in sub-optimal quality-latency trade-offs. Unlike machines, human interpreters leverage their professional expertise to perceive and organize source speech into meaningful “interpreting units” before initiating translation, thereby achieving an effective balance between translation quality and latency. Recent state-of-the-art systems formulate SimulST as a multi-turn dialogue task, mimicking human interpreters by leveraging LLMs to identify interpreting units and make read/write decisions. However, using LLMs to make read/write decisions inevitably incurs high computational overhead, particularly in settings that require frequent decision-making. This inefficiency renders them unsuitable for real-world low-latency scenarios and motivates the development of a more efficient alternative. In this paper, we present a novel SimulST framework, SimulSense. Motivated by the interpreting-unit-based decision process of human interpreters, SimulSense dynamically senses interpreting units and governs read/write decisions via a lightweight Units Detector module, which enables efficient decision-making by decoupling this process from the heavy LLM translation backbone. Experiments across four SimulST settings and multiple language pairs demonstrate that SimulSense achieves strong quality-latency trade-offs and substantially improves decision-making efficiency. Furthermore, we show that SimulSense approaches the upper range of human-level latency in realistic unbounded scenarios for several language pairs, highlighting its potential for real-world low-latency deployment.