New federated learning approach cuts communication costs by varying update frequencies across model layers

A new federated learning algorithm reduces communication overhead while maintaining—or even improving—model performance by applying different synchronization schedules to different parts of neural networks.

Researchers analyzing how federated learning systems can generalize better have found that not all model components need to be updated with equal frequency. Their theoretical analysis reveals that intermediate representation layers benefit from less frequent synchronization compared to the output layers, especially when training data is unevenly distributed across clients.

The Problem with Standard Federated Learning#

Federated learning trains machine learning models across distributed devices or organizations without centralizing raw data. However, this comes at a cost: constant communication between clients and a central server to synchronize model updates consumes significant bandwidth and energy, particularly in scenarios with non-independent and identically distributed (non-iid) data—a common real-world situation where different participants have fundamentally different datasets.

Theoretical Insights#

The researchers developed tighter mathematical bounds on how well federated models generalize, accounting for both the variation in data across clients and the number of local training steps performed before aggregation. This theoretical framework suggests that representation-learning components—typically earlier network layers that extract features—can safely update less frequently without harming overall accuracy. Meanwhile, task-specific layers at the end of the network benefit from more frequent synchronization.

The FedALS Algorithm#

Building on these insights, the team designed Federated Learning with Adaptive Local Steps (FedALS), which uses variable aggregation frequencies for different model sections. Representation extractors synchronize less often, reducing the total communication burden, while output layers maintain more frequent updates. Experimental validation demonstrates that this differentiated approach decreases communication costs without sacrificing—and sometimes improving—model generalization, particularly in challenging non-iid settings.

The work is accompanied by publicly available code to support reproducibility and adoption.