Zoe Liu, Wenyao Liu

The demand for high-quality, low-latency, high-density live streaming is rapidly growing, driven by content such as sports, gaming, and real-time events. This talk focuses on GPU-based, AI-driven video compression solutions, particularly for infrastructures already utilizing NVIDIA’s NVENC hardware (HW) encoder, which offers high density and low compute cost for live transcoding. GPU-based HW encoding outperforms CPU-based software solutions w.r.t. density and latency. However, it often falls short in visual quality and bitrate efficiency. We explore an integrated solution that combines NVENC with GPU-based AI algorithms to build an intelligent, high-quality, and bitrate-efficient live transcoding pipeline. We will present a practical approach that integrates NVENC HW encoding with ML-based video processing. With available GPU resources, advanced neural networks can handle no-reference video quality assessment (VQA), content-aware pre-analysis, adaptive enhancement, and region-of-interest (ROI) detection. Specifically, we will present VQA algorithms that generate an overall quality score (0–100), along with 10 sub-scores (0–1), each capturing unique video characteristics, such as blockiness, blur, noise, exposure, contrast, and motion complexity. Enhancement techniques like tone mapping, denoising, and color preservation will then be applied guided by VQA scores. ROI detection is crucial for optimized bitrate allocation and mode selection. In sports streams, for instance, perceptually important areas such as faces or jersey numbers should be prioritized. Less critical areas like grass can be more heavily compressed. However, consistent compression across frames helps preserve visual stability. We focus on the practical realization of AI-enhanced real-time video delivery, targeted on three key objectives: (1) achieving sufficient density on par with existing NVENC performance to ensure compute efficiency; (2) maintaining low latency for live use cases; and (3) leveraging AI algorithms and NVENC APIs to enhance visual quality while reducing bitrate. We will share quantitative results and real-world use cases to verify the effectiveness of our proposed solutions.