2017–2020
Foundations
2017 · Attention is All You Need
Milestones
The Transformer architecture revolutionizes NLP
New Capabilities Unlocked
Language Understanding
Parse and comprehend human language structure, meaning, and context
Language
Impact
Translation services became dramatically better. Google Translate and other tools could now understand context, making translations more natural and accurate. This made global communication easier for everyone.
- Parameters
- ~65M
- Context
- 512 tokens
Source Vaswani et al., 2017
Key terms
- Transformer
- Neural network architecture using self-attention to process sequences in parallel, replacing older recurrent models.
- Tokens
- Pieces of text the model processes. A token is roughly 4 characters or 0.75 words in English.
- Parameters
- Learned weights in the model. More parameters generally means more knowledge and capability, but also higher cost.
- Few-Shot Learning
- Learning to perform a task from just a few examples provided in the prompt, without retraining.