X-ray prohibited item detection (XPID) aims to automatically recognize prohibited items and localize them in X-ray security images, which has recently received increasing attention. However, most existing methods adopt a closed-set assumption, thereby restricting detection to the classes predefined during training. In real-world security inspection, such fixed training classes cannot adequately capture the diversity and continual emergence of prohibited items. To address this limitation, we investigate open-set X-ray prohibited item detection (OXPID), which trains models only on known classes while enabling them to detect both known threats and previously unseen prohibited items. Specifically, we introduce two key modules tailored to the complex visual characteristics of X-ray images in the open-set detection. The uncertainty-aware unknown discovery (U2D) module employs an unsupervised strategy to mine pseudo-unknown samples from high-uncertainty regions, thereby facilitating the discovery of unknown items. Moreover, it integrates an evidential deep learning-based loss that provides principled supervision for pseudo-unknown training while mitigating two inherent forms of supervision conflict. In parallel, the positive-negative prototype learning (PNPL) module leverages both positive prototypes and reciprocal negative prototypes to construct a discriminative embedding space. By reinforcing the attraction of features toward positive prototypes and their repulsion from negative prototypes, the PNPL module encourages compact representations of known classes while refining the decision boundary between known and unknown prohibited items. Extensive experiments demonstrate that our method achieves state-of-the-art performance on the two customized OXPID benchmarks and exhibits strong generalization capability when evaluated on an RGB-based open-set object detection benchmark. The source code and dataset are publicly available at https://github.com/HuangLian126/OXPID
