XAI-Driven Malicious Encrypted Traffic Detection and Characterization to Enhance Information Security

Abstract

Securing information through encryption is essential in data communication, but to effectively detect malicious activities, it is crucial to distinguish between encrypted and non-encrypted traffic. Traditional encrypted traffic classification methods, including rule-based systems and conventional machine learning approaches, often struggle with scalability, generalization, and class imbalance, leading to suboptimal classification performance. This study introduces a novel hybrid model for encrypted traffic classification, integrating Multi-Head Attention mechanisms for feature enhancement and LightGBM as the final classifier. The proposed model follows a two-step classification process: first, performing binary classification to separate encrypted and non-encrypted traffic, and second, applying multi-class classification to categorize encrypted traffic into TOR, VPN, I2P, Zeronet, and Freenet. To improve model interpretability, SHAP is employed to validate the importance of attention-based features, while LIME provides insights into misclassified instances, enabling adjustments such as weight threshold tuning and handling class imbalances.

Furthermore, this study incorporates a refined dataset preprocessing pipeline, leveraging NTL Flowlyzer—an advanced traffic analyzer that extracts over 400 features, including entropy-based attributes. To address class imbalance issues, strategic adjustments such as SMOTE augmentation for Freenet and class-specific threshold tuning were applied based on SHAP and LIME insights, resulting in improved classification performance. The experimental evaluation demonstrates that the proposed hybrid model outperforms existing approaches in accuracy, precision ,and recall while maintaining efficiency in both time and computational complexity. By integrating explainable AI techniques and adaptive optimization strategies, our approach enhances classification performance and improves the transparency and interpretability of encrypted traffic detection. These findings contribute to advancing cybersecurity by enabling more robust and interpretable encrypted traffic classification models.