Dynamic Forward and Backward Sparse Training (DFBST): Accelerated Deep Learning through Completely Sparse Training Schedule
Published in Asian Conference on Machine Learning, 2022
In this work, we address the shortcomings of existing dynamic sparse training methods aimed at finding sparse subnetworks within deep neural networks during training.We introduce DFBST, a novel sparse training algorithm which sparsifies both weights and gradients of the network using trainable thresholds which are jointly optimized with the network parameters through the same loss function.