Your new post is loading...
"Posted by Ehsan Amid, Research Scientist, and Rohan Anil, Principal Engineer, Google Research, Brain Team"
|
Scooped by Juliette Decugis |
Scoop.it!
Juliette Decugis's insight:
Many recent ML papers hope to address the overwhelming problem of deep learning models: their computational and memory cost. Whereas lots of recent work has focused on the sparsification of said models, LocoProp attempts to rethink backpropagation - the most expensive step of neural network training.
LocoProp decomposes a model's objective function into a layer-wise loss, comparing a layer's output and the overall bath's final output, accompanied by a regularizer term (L2 loss). Breaking down the loss function across layers permits parallelization of training, smaller order calculations and more flexibility. Furthermore, the paper demonstrates that "the overall behavior of the combined updates closely resembles higher-order updates."
Potential limits of the paper: "small" networks used, "still remains to be seen how well the method generally works across tasks."
Link to the complete paper: https://proceedings.mlr.press/v151/amid22a/amid22a.pdf
No comment yet.
Sign up to comment
