◑ applying · kind concept · level 3 · 12h
- Generaliza: Self-Attention
Runs several attention heads in parallel, each learning a different relation, then concatenates them. More representational capacity at the same asymptotic cost.
In machine learning, Attention is a method that determines the importance of each component in a sequence relative to the other components in that sequence. In Natural Language Processing, importance is represented by “soft” weights assigned to each word in a sentence. More generally, Attention encodes vectors called token Embeddings across a fixed-width sequence that can range from tens to millions of tokens in size.
Enlaces
- Generaliza: Self-Attention