Pavlo Molchanov, Research Director at NVIDIA

Vision-language models are emerging as a new generalized interface for image processing, moving the field beyond separate pipelines for captioning, retrieval, recognition, and reasoning toward unified visual systems. These models consist of two core components: a language model for reasoning and a vision backbone that converts multimedia inputs into representations the language model can interpret. Vision backbones, or image foundation models, such as CLIP, DINOv2, SAM, and RADIO are shifting research and development toward generalized visual backbones that can support a wide range of image processing tasks with few-shot adaptation or minimal fine-tuning, while reducing the need for task-specific preprocessing. In this talk, I will discuss recent progress and remaining challenges in building such open models, with an emphasis on training and deployment efficiency. I will also highlight what remains unsolved for real-world deployment in robotics, autonomous vehicles, and general computer vision, including robustness, controllability, grounding, efficiency, and evaluation beyond closed benchmarks. The broader goal is to position VLMs not merely as a multimodal trend, but as a serious foundation for the next generation of generalized image processing systems.