Neural network design based on self-attention layers that scales with data and compute—foundation of modern LLMs and many multimodal models.
Transformers process token sequences with attention: each position can attend to others so the model mixes long-range dependencies more flexibly than older RNNs. “Attention is all you need” is the famous origin; in practice you stack blocks with feed-forward layers, norms, and residuals.
Why managers and engineers care
You do not need matrix calculus daily, but the mental model explains costs: sequence length drives memory for KV caches, and width/depth drives latency and price at inference time.