Contributed by Hossein Ebrahimi Dinaki, Shervin Shirmohammadi, Emil Janulewicz, and David Côté, based on the  IEEEXplore® article “Forecasting Video QoE with Deep Learning from Multivariate Time-series”, IEEE Open Journal of Signal Processing, Vol. 2, 2021, pp. 512-521.

Video streaming systems are designed to react to current network and player conditions, but reacting alone leaves users vulnerable to quality degradations that unfold over time. A proactive approach to video Quality of Experience (QoE) forecasting is better because it gives streaming systems time to intervene before users actually perceive a problem. Instead of waiting for buffering, bitrate drops, or quality fluctuations to occur and then correcting them, forecasting can anticipate future QoE degradations and trigger earlier actions such as bitrate adaptation, prefetching, or buffer adjustment. This shift from response to prediction is especially valuable because user experience often depends on short-lived but highly noticeable events, and even brief quality disruptions can have a disproportionate impact on satisfaction. Due to these reasons, other works have also looked into video QoE forecasting ‎[1] ‎[2] .

Our work is one of the earlier proposals that takes this proactive approach and reframes video QoE as a forecasting problem, demonstrating that combining temporal and local features extracted from QoE metrics yields a more accurate next-step QoE forecast than reactive approaches.

QoE forecasting with machine learning for signal processing

Treating QoE as a multivariate time series driven by interacting quality metrics connects multimedia quality assessment to classical signal-processing themes such as time-series modeling and sequential prediction. We propose a two-part architecture that reframes QoE prediction as a multivariate time-series forecasting problem. The model takes six inputs: playback duration, buffering length, buffering frequency, average bitrate, happiness score, and the calculated total QoE. The first part is a feature-extraction module that learns from these inputs how quality has evolved from past observations and recent measurements. The second part is a forecasting module that uses those features to predict the QoE at the next step. This approach directly addresses temporal dependency and nonlinear feature interaction, a common challenge in many signal processing systems where the relevant features evolve over time.

What the method achieves

We evaluate the framework on a multivariate time-series QoE dataset collected from an industry video-streaming testbed and compare it to four well-known baselines: a classic forecasting model (SVR), a basic neural network (MLP), and two strong sequence models (LSTM and BiLSTM). Results show reduced prediction error and higher correlation with ground-truth QoE across all six forecasted metrics. This demonstrates that our hybrid BiLSTM-CNN approach successfully captures both the long-term temporal memory and the complex spatial interactions of the metrics, outperforming the baseline models by jointly modeling temporal dependency and local spatial structure. We also compare SELU and tanh activation functions , showing each component's contribution across three different LSTM-based models which is finding SELU improves accuracy for most metrics.

                                        Fig 2. Evaluation Results

Practical impact

For video streaming engineers, accurate next-step QoE forecasts enable proactive control: bitrate selection, prefetching, adaptive buffering, and resource allocation that anticipate user-perceived degradation rather than respond after the fact. For researchers, framing QoE as a multivariate time-series forecasting problem invites extensions such as personalization, multimodal context (e.g., device, content type), and uncertainty-aware predictions that communicate confidence to downstream controllers. By recasting QoE prediction as multivariate time-series forecasting, we bridge traditional streaming-quality metrics with deep sequential learning methods.

Summary takeaways

• Integrate next-step QoE forecasting into adaptive streaming pipelines to shift from reactive to proactive quality management.
• Prioritize model components that capture temporal dependency and nonlinear feature interaction when forecasting the next-step QoE. 

References

1. M. Petrou, D. Pradas, M. Royer and E. Lochin, "Forecasting YouTube QoE over SATCOM," IEEE Vehicular Technology Conference, Florence, Italy, 2023, pp. 1-5. DOI: 10.1109/VTC2023-Spring57618.2023.10200133
2. Z. Zhou, Y. Dong, L. Song, R. Xie, L. Li and B. Zhou, "Quality of Experience Evaluation for Streaming Video Using CGNN," IEEE International Conference on Visual Communications and Image Processing (VCIP), Macau, China, 2020, pp. 285-288. DOI: 10.1109/VCIP49819.2020.9301799