EVSR: Expert-Driven Visibility-Aware Selective Representation Learning From Foundation Vision Models for Remote Sensing Semantic Segmentation

Vision foundation models have recently emerged as a powerful paradigm for visual understanding by learning transferable representations from large-scale unlabeled data. While self-supervised vision transformers such as DINOv3 exhibit remarkable generalization across natural image benchmarks, their direct deployment in remote sensing (RS) semantic segmentation remains challenging due to pronounced domain discrepancies in viewing geometry, spatial resolution, illumination conditions, and surface structure, which often lead to hierarchical representation misalignment and limited transferability of semantic knowledge. To address these challenges, this paper presents EVSR, an expert-driven, visibility-aware selective representation learning framework for RS semantic segmentation built upon foundation vision representations. Instead of shallow adaptation, EVSR performs hierarchical adaptation of self-supervised representations via block-wise representation alignment, enabling task-driven refinement across multiple semantic depths while preserving general visual priors and ensuring efficient and stable transfer. Furthermore, we introduce a visibility-aware selective attention mechanism on multi-scale foundation representations. By applying multiple visibility masks to induce different perception levels, we creatively interpret the multi-visible attention modules as specialized experts and employ a mixture-of-experts aggregation to adaptively integrate their responses, yielding selective and expert-guided representation refinement across semantic granularities. Extensive experiments on three benchmarks demonstrate that the proposed EVSR consistently outperforms state-of-the-art methods. Specifically, EVSR achieves mIoU scores of 87.38%, 84.78%, and 49.87% on the ISPRS Potsdam, Vaihingen, and MMHunan datasets, respectively, demonstrating strong adaptability across diverse RS benchmarks with varying spatial resolutions and scene complexities. The source code is accessible at https://github.com/sstary/SSRS.