Intrusion Detection System for Open Network Scenarios: A Known/Unknown Attack Detection Method Based on Inter-Class Relationships

Intrusion detection systems (IDS) are crucial for protecting the security of network devices. In a dynamic network environment, network attack behavior is constantly changing. IDS developed using closed datasets has limited detection capabilities and cannot effectively respond to potential unknown attacks. Existing methods exhibit poor generalization to unknown attacks and have rigid decision boundaries that struggle to adapt to evolving network attacks. To address this challenge, we propose a known/unknown attack detection method based on inter-class relationships with the collaboration of virtual class and soft labels, which includes: 1) feature space optimization; and 2) quantification of uncertainty for unknown attack detection. Firstly, to construct clear decision boundaries in high-dimensional space and reserve representation space for unknown attacks, we design to adaptively insert a virtual class in the neighborhood of known attack categories, reducing the intra-class distance of known attacks and expanding the inter-class intervals. Secondly, we learn soft-label prototypes from known samples to characterize the similarity and difference relationships among known attack categories. We quantify classification uncertainty by computing the Kullback-Leibler divergence between these prototypes and the predicted outputs, after which a Gaussian Mixture Model is applied to model the divergence distribution and adaptively identify unknown attacks. Extensive experimental results on CICIDS2017, NSL-KDD, and CICIoV2024 demonstrate that the proposed method achieves superior performance across various evaluation metrics. Compared with state-of-the-art methods, our approach improves the harmonic mean of known and unknown class detection accuracy by over 7% on NSL-KDD, while specifically boosting the unknown attack detection accuracy by approximately 1.12%, 5.24% on the CICIDS2017, NSL-KDD datasets, respectively.