An MQTT Application-Layer Traffic Analyzer for Interpretable Flow-Level Intrusion Detection and Zero-DayThreat Identification in IoT Environment Using TabNet

Abstract

Message Queuing Telemetry Transport (MQTT) is widely used in IoT systems; however, its lightweight design makes it vulnerable to various cyberattacks. This research reviews existing intrusion detection methods for MQTT and shows their limitations in detecting new and complex threats. This study presents a comprehensive intrusion detection framework that uses raw PCAP data and employs flow-based behavioral analysis to detect both known and novel attacks. We present MQTTFlowLyzer, a protocol-aware analyzer designed to extract detailed MQTT flow features and generate an augmented dataset, BCCC-MQTT-IDS-2025, that captures realistic and diverse attack scenarios. The extracted features train a TabNet-based learning model capable of integrated feature selection, classification, and confidence-based detection of zero-day threats. Our approach highlights the behavioral uniqueness of each attack class and uses attention-driven interpretability for in-depth analysis. Experimental results demonstrate that the model effectively detects attacks while maintaining high performance across other categories. The system successfully flags previously unseen traffic by profiling class-specific behaviors and incorporating confidence thresholds. These results demonstrate the potential of flow-based, interpretable learning for real-time and resilient MQTT intrusion detection.