Federated Learning for Heterogeneous Networks: Algorithmic and System Design

Building reliable machine learning models depends on access to data samples. With the increasingly advanced sensing and computing capabilities on edge devices, the ever-stringent data privacy legislation, and growing user privacy concerns, it is crucial to build learning models from separate, heterogeneous data sources without violating user privacy. Federated Learning (FL) can facilitate collaborative machine learning without accessing user-sensitive data and has emerged as an attractive paradigm for mobile edge networks. However, federated optimization builds on a heterogeneous environment, which brings challenges beyond traditional distributed learning. Though FL is viewed as a promising technique for enabling intelligent applications, the current FL system suffers from high communication costs, restricting it from being applied in mobile edge networks. To fully release the potential, the FL design must be communication-efficient, adaptive, and robust to the heterogeneous training environment.

In this thesis, we aim to address the practical challenges of FL in a conscientious manner. Particularly, we try to understand and address some of those challenges in federated networks and build FL systems that fulfill the accuracy, efficiency, and robustness requirements. Starting with the primary challenge, i.e., data heterogeneity, we study how it impacts the model accuracy and communication cost in the collaborative training system. To address this concern, we develop new and scalable algorithms that can quantify the contribution from participating devices, thus alleviating the negative impact of data heterogeneity and reducing the overall communication burden. To handle another major challenge, i.e., the heterogeneity of computation capabilities among different types of edge devices, we devise a new sub-model training method to enable devices with heterogeneous computation capabilities to participate in and contribute to the FL system, making it robust to the straggler effect. The proposed solutions are rigorously compared with popularly adopted benchmarks from theoretical and empirical perspectives. Finally, we provide a preliminary discussion on personalized FL and point out the potentially interesting research directions in the related fields. Although the proposed methods and designs originate from the practical application of FL, the theoretical insights gained from this thesis can be extended to a broader context of trustworthy machine learning.