XAI-Driven Malicious Encrypted Traffic Detection And Characterization To Enhance Information Security

Abstract

Securing information through encryption is essential in data communication, but to effectively detect maliciousactivities, it is crucial to distinguish between encrypted and non-encrypted traffic. Traditional encrypted trafficclassification methods, including rule-based systems and conventional machine learning approaches, often strugglewith scalability, generalization, and class imbalance, leading to suboptimal classification performance. This studyintroduces a novel hybrid model for encrypted traffic classification, integrating Multi-Head Attention mechanismsfor feature enhancement and LightGBM as the final classifier. The proposed model follows a two-step classificationprocess: first, performing binary classification to separate encrypted and non-encrypted traffic, and second, applyingmulti-class classification to categorize encrypted traffic into TOR, VPN, I2P, Zeronet, and Freenet. To improvemodel interpretability, SHAP is employed to validate the importance of attention-based features, while LIMEprovides insights into misclassified instances, enabling adjustments such as weight threshold tuning and handlingclass imbalances.Furthermore, this study incorporates a refined dataset preprocessing pipeline, leveraging NTL Flowlyzer—anadvanced traffic analyzer that extracts over 400 features, including entropy-based attributes. To address classimbalance issues, strategic adjustments such as SMOTE augmentation for Freenet and class-specific threshold tuningwere applied based on SHAP and LIME insights, resulting in improved classification performance. The experimentalevaluation demonstrates that the proposed hybrid model outperforms existing approaches in accuracy, precision,and recall while maintaining efficiency in both time and computational complexity. By integrating explainable AItechniques and adaptive optimization strategies, our approach enhances classification performance and improves thetransparency and interpretability of encrypted traffic detection. These findings contribute to advancing cybersecurityby enabling more robust and interpretable encrypted traffic classification models.