Attention? Attention!
📰 Lilian Weng's Blog
Attention mechanisms in deep learning are inspired by how humans focus on specific parts of an image or sentence
Action Steps
- Understand the motivation behind attention mechanisms
- Learn about different types of attention, such as self-attention and soft vs hard attention
- Explore various attention-based models, including Transformer and Pointer Network
Who Needs to Know This
ML researchers and engineers can benefit from understanding attention mechanisms to improve model performance, especially in NLP and computer vision tasks
Key Insight
💡 Attention mechanisms allow models to focus on specific parts of the input data, enhancing their ability to learn complex patterns
Share This
💡 Attention mechanisms in deep learning mimic human visual attention, improving model performance in NLP & CV tasks
Key Takeaways
Attention mechanisms in deep learning are inspired by how humans focus on specific parts of an image or sentence
Full Article
Published Time: 2018-06-24T00:00:00Z
# Attention? Attention! | Lil'Log
[Lil'Log](https://lilianweng.github.io/ "Lil'Log (Alt + H)")
* |
* [Posts](https://lilianweng.github.io/ "Posts")
* [Archive](https://lilianweng.github.io/archives "Archive")
* [Search](https://lilianweng.github.io/search/ "Search (Alt + /)")
* [Tags](https://lilianweng.github.io/tags/ "Tags")
* [FAQ](https://lilianweng.github.io/faq "FAQ")
# Attention? Attention!
Date: June 24, 2018 | Estimated Reading Time: 21 min | Author: Lilian Weng
Table of Contents
* [What’s Wrong with Seq2Seq Model?](https://lilianweng.github.io/posts/2018-06-24-attention/#whats-wrong-with-seq2seq-model)
* [Born for Translation](https://lilianweng.github.io/posts/2018-06-24-attention/#born-for-translation)
* [Definition](https://lilianweng.github.io/posts/2018-06-24-attention/#definition)
* [A Family of Attention Mechanisms](https://lilianweng.github.io/posts/2018-06-24-attention/#a-family-of-attention-mechanisms)
* [Summary](https://lilianweng.github.io/posts/2018-06-24-attention/#summary)
* [Self-Attention](https://lilianweng.github.io/posts/2018-06-24-attention/#self-attention)
* [Soft vs Hard Attention](https://lilianweng.github.io/posts/2018-06-24-attention/#soft-vs-hard-attention)
* [Global vs Local Attention](https://lilianweng.github.io/posts/2018-06-24-attention/#global-vs-local-attention)
* [Neural Turing Machines](https://lilianweng.github.io/posts/2018-06-24-attention/#neural-turing-machines)
* [Reading and Writing](https://lilianweng.github.io/posts/2018-06-24-attention/#reading-and-writing)
* [Attention Mechanisms](https://lilianweng.github.io/posts/2018-06-24-attention/#attention-mechanisms)
* [Pointer Network](https://lilianweng.github.io/posts/2018-06-24-attention/#pointer-network)
* [Transformer](https://lilianweng.github.io/posts/2018-06-24-attention/#transformer)
* [Key, Value and Query](https://lilianweng.github.io/posts/2018-06-24-attention/#key-value-and-query)
* [Multi-Head Self-Attention](https://lilianweng.github.io/posts/2018-06-24-attention/#multi-head-self-attention)
* [Encoder](https://lilianweng.github.io/posts/2018-06-24-attention/#encoder)
* [Decoder](https://lilianweng.github.io/posts/2018-06-24-attention/#decoder)
* [Full Architecture](https://lilianweng.github.io/posts/2018-06-24-attention/#full-architecture)
* [SNAIL](https://lilianweng.github.io/posts/2018-06-24-attention/#snail)
* [Self-Attention GAN](https://lilianweng.github.io/posts/2018-06-24-attention/#self-attention-gan)
* [References](https://lilianweng.github.io/posts/2018-06-24-attention/#references)
[Updated on 2018-10-28: Add [Pointer Network](https://lilianweng.github.io/posts/2018-06-24-attention/#pointer-network) and the [link](https://github.com/lilianweng/transformer-tensorflow) to my implementation of Transformer.]
[Updated on 2018-11-06: Add a [link](https://github.com/lilianweng/transformer-tensorflow) to the implementation of Transformer model.]
[Updated on 2018-11-18: Add [Neural Turing Machines](https://lilianweng.github.io/posts/2018-06-24-attention/#neural-turing-machines).]
[Updated on 2019-07-18: Correct the mistake on using the term “self-attention” when introducing the [show-attention-tell](https://arxiv.org/abs/1502.03044) paper; moved it to [Self-Attention](https://lilianweng.github.io/posts/2018-06-24-attention/#self-attention) section.]
[Updated on 2020-04-07: A follow-up post on improved Transformer models is [here](https://lilianweng.github.io/posts/2020-04-07-the-transformer-family/).]
Attention is, to some extent, motivated by how we pay visual attention to different regions of an image or correlate words in one sentence. Take the picture of a Shiba Inu in Fig. 1 as an example.
A Shiba Inu in a men’s outfit. The credit of the original photo goes
# Attention? Attention! | Lil'Log
[Lil'Log](https://lilianweng.github.io/ "Lil'Log (Alt + H)")
* |
* [Posts](https://lilianweng.github.io/ "Posts")
* [Archive](https://lilianweng.github.io/archives "Archive")
* [Search](https://lilianweng.github.io/search/ "Search (Alt + /)")
* [Tags](https://lilianweng.github.io/tags/ "Tags")
* [FAQ](https://lilianweng.github.io/faq "FAQ")
# Attention? Attention!
Date: June 24, 2018 | Estimated Reading Time: 21 min | Author: Lilian Weng
Table of Contents
* [What’s Wrong with Seq2Seq Model?](https://lilianweng.github.io/posts/2018-06-24-attention/#whats-wrong-with-seq2seq-model)
* [Born for Translation](https://lilianweng.github.io/posts/2018-06-24-attention/#born-for-translation)
* [Definition](https://lilianweng.github.io/posts/2018-06-24-attention/#definition)
* [A Family of Attention Mechanisms](https://lilianweng.github.io/posts/2018-06-24-attention/#a-family-of-attention-mechanisms)
* [Summary](https://lilianweng.github.io/posts/2018-06-24-attention/#summary)
* [Self-Attention](https://lilianweng.github.io/posts/2018-06-24-attention/#self-attention)
* [Soft vs Hard Attention](https://lilianweng.github.io/posts/2018-06-24-attention/#soft-vs-hard-attention)
* [Global vs Local Attention](https://lilianweng.github.io/posts/2018-06-24-attention/#global-vs-local-attention)
* [Neural Turing Machines](https://lilianweng.github.io/posts/2018-06-24-attention/#neural-turing-machines)
* [Reading and Writing](https://lilianweng.github.io/posts/2018-06-24-attention/#reading-and-writing)
* [Attention Mechanisms](https://lilianweng.github.io/posts/2018-06-24-attention/#attention-mechanisms)
* [Pointer Network](https://lilianweng.github.io/posts/2018-06-24-attention/#pointer-network)
* [Transformer](https://lilianweng.github.io/posts/2018-06-24-attention/#transformer)
* [Key, Value and Query](https://lilianweng.github.io/posts/2018-06-24-attention/#key-value-and-query)
* [Multi-Head Self-Attention](https://lilianweng.github.io/posts/2018-06-24-attention/#multi-head-self-attention)
* [Encoder](https://lilianweng.github.io/posts/2018-06-24-attention/#encoder)
* [Decoder](https://lilianweng.github.io/posts/2018-06-24-attention/#decoder)
* [Full Architecture](https://lilianweng.github.io/posts/2018-06-24-attention/#full-architecture)
* [SNAIL](https://lilianweng.github.io/posts/2018-06-24-attention/#snail)
* [Self-Attention GAN](https://lilianweng.github.io/posts/2018-06-24-attention/#self-attention-gan)
* [References](https://lilianweng.github.io/posts/2018-06-24-attention/#references)
[Updated on 2018-10-28: Add [Pointer Network](https://lilianweng.github.io/posts/2018-06-24-attention/#pointer-network) and the [link](https://github.com/lilianweng/transformer-tensorflow) to my implementation of Transformer.]
[Updated on 2018-11-06: Add a [link](https://github.com/lilianweng/transformer-tensorflow) to the implementation of Transformer model.]
[Updated on 2018-11-18: Add [Neural Turing Machines](https://lilianweng.github.io/posts/2018-06-24-attention/#neural-turing-machines).]
[Updated on 2019-07-18: Correct the mistake on using the term “self-attention” when introducing the [show-attention-tell](https://arxiv.org/abs/1502.03044) paper; moved it to [Self-Attention](https://lilianweng.github.io/posts/2018-06-24-attention/#self-attention) section.]
[Updated on 2020-04-07: A follow-up post on improved Transformer models is [here](https://lilianweng.github.io/posts/2020-04-07-the-transformer-family/).]
Attention is, to some extent, motivated by how we pay visual attention to different regions of an image or correlate words in one sentence. Take the picture of a Shiba Inu in Fig. 1 as an example.
A Shiba Inu in a men’s outfit. The credit of the original photo goes
DeepCamp AI