Efficient sparse collective communication and its application to accelerate distributed deep learning

Jiawei Fei, Chen-Yu Ho, Atal N. Sahu, Marco Canini, Amedeo Sapio

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Efficient collective communication is crucial to parallel-computing applications such as distributed training of large-scale recommendation systems and natural language processing models. Existing collective communication libraries focus on optimizing operations for dense inputs, resulting in transmissions of many zeros when inputs are sparse. This counters current trends that see increasing data sparsity in large models. We propose OmniReduce, an efficient streaming aggregation system that exploits sparsity to maximize effective bandwidth use by sending only non-zero data blocks. We demonstrate that this idea is beneficial and accelerates distributed training by up to 8.2x. Even at 100 Gbps, OmniReduce delivers 1.4--2.9x better performance for network-bottlenecked DNNs.
Original languageEnglish (US)
Title of host publicationProceedings of the 2021 ACM SIGCOMM 2021 Conference
PublisherACM
ISBN (Print)9781450383837
DOIs
StatePublished - Sep 30 2020

Fingerprint

Dive into the research topics of 'Efficient sparse collective communication and its application to accelerate distributed deep learning'. Together they form a unique fingerprint.

Cite this