Anomaly Detection and Attack Mitigation in Federated Learning

Abstract

Federated Learning (FL) enables collaborative model training across multiple participants without sharing raw data, achieving a balance between privacy protection and data utility. However, the decentralized nature of FL also introduces new security threats, among which model poisoning attacks are the most critical. Malicious clients can upload manipulated model updates to disrupt the global aggregation process, leading to performance degradation or even system failure. This thesis systematically investigates anomaly detection (AD) and attack mitigation in FL, aiming to enhance system robustness and security from both detection and defense perspectives. First, a comprehensive review and categorization of AD methods are presented, focusing on reconstruction-based and prediction-based deep learning frameworks and their applications to different data types. Second, existing defense mechanisms are analyzed in terms of their effectiveness and limitations under various attack scenarios, providing the theoretical foundation for the proposed framework. Based on these analyses, this thesis proposes an unsupervised defense framework named Dual-VAE with Truncated Gaussian (DVTG). The framework follows a three-stage structure to model and filter client updates. In Stage 1, a variational autoencoder (VAE) is trained to estimate reconstruction errors and identify a set of potentially clean updates. In Stage 2, a second VAE with a truncated Gaussian prior is trained on this refined subset to obtain a more stable latent representation. In Stage 3, the trained model evaluates incoming client updates and filters those with high reconstruction errors before aggregation. The method enables effective anomaly detection without requiring labeled or clean data and remains stable under both adversarial and stochastic disturbances. Experiments conducted on the MNIST dataset under non-independent and identically distributed (non-IID) conditions show that DVTG outperforms the baseline model across different attack scenarios. The framework effectively detects malicious clients while maintaining stable convergence and comparable accuracy to the non-attack scenario. Finally, this thesis discusses several future directions. These include extending the defense framework to hierarchical FL architectures and developing more interpretable and efficient AD models. The goal is to build a reliable and practical FL framework with stronger defense capability and better adaptability to real-world environments.