arXiv Analytics

Sign in

arXiv:2006.16236 [cs.LG]AbstractReferencesReviewsResources

Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention

Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, Fran├žois Fleuret

Published 2020-06-29Version 1

Transformers achieve remarkable performance in several tasks but due to their quadratic complexity, with respect to the input's length, they are prohibitively slow for very long sequences. To address this limitation, we express the self-attention as a linear dot-product of kernel feature maps and make use of the associativity property of matrix products to reduce the complexity from $\mathcal{O}\left(N^2\right)$ to $\mathcal{O}\left(N\right)$, where $N$ is the sequence length. We show that this formulation permits an iterative implementation that dramatically accelerates autoregressive transformers and reveals their relationship to recurrent neural networks. Our linear transformers achieve similar performance to vanilla transformers and they are up to 4000x faster on autoregressive prediction of very long sequences.

Comments: ICML 2020, project at
Categories: cs.LG, stat.ML
Related articles: Most relevant | Search more
arXiv:2001.11771 [cs.LG] (Published 2020-01-31)
Encoding-based Memory Modules for Recurrent Neural Networks
arXiv:1802.00150 [cs.LG] (Published 2018-02-01)
Alternating Multi-bit Quantization for Recurrent Neural Networks
arXiv:1902.07275 [cs.LG] (Published 2019-02-19)
Understanding and Controlling Memory in Recurrent Neural Networks