The effectiveness of vision token pruning critically depends on the criterion used to define token redundancy. Existing approaches, which primarily rely on global saliency scores or similarity heuristics, often fail to account for a token’s essential contribution to the model’s internal representations. This oversight results in retaining visually prominent but redundant tokens or merging semantically distinct features, severely degrading performance under tight compression budgets. To address this, we propose a novel token pruning strategy aligned with representation complexity. We first introduce Local Representation Complexity (LRC), a token-level measure that estimates the intrinsic dimension of each token’s embedding. Tokens exhibiting high LRC are considered locally unpredictable and are therefore prioritized for preservation. To ensure selection diversity and mitigate over-concentration in the selected set, we frame the pruning process as a budget allocation problem and introduce a Greedy Selection algorithm with Penalized Diversity to actively suppress redundancy. We evaluate the proposed method across five vision-language model backbones from three model families, including LLaVA-1.5/LLaVA-NeXT, Qwen2.5-VL/Qwen3-VL, and InternVL3, covering a wide range of capacities and tasks. The results demonstrate competitive performance across different backbones, with particularly notable gains at aggressive compression ratios.
