⌘K
ENES
GitHub
conceptLanguage & Foundation Models~1 min readUpdated 2026-06-07#llms#transformers#attention

Transformer Attention Map

Attention lets a token representation read from other token representations. It is not magic memory; it is a learned routing pattern over the current context.

Why It Matters

  • It explains why context order and formatting affect model behavior.
  • It helps separate retrieval problems from reasoning or instruction problems.
  • It gives a mental model for context limits, compression, and prompt structure.

Practical Check

When an LLM ignores a detail, ask whether the detail was visible, salient, and connected to the task. Then test with a smaller, sharper context before changing models.

Related notes

Language & Foundation ModelsThe decoder transformerHow the GPT-style architecture assembles attention, feed-forward layers, residuals, and norms into a causal next-token predictor.Language & Foundation ModelsBase vs instruct vs chat modelsA base model completes text; an instruct/chat model follows requests. The post-training stack (SFT + preference alignment) that turns one into the other.Language & Foundation ModelsContext window & the KV cacheThe context window is the model's working memory; the KV cache is what makes generation fast. Both explain why long prompts cost more and why latency behaves as it...Language & Foundation ModelsDecoding & samplingA model outputs a probability distribution; decoding turns it into text. Temperature, top-p/top-k, greedy vs sampling — the knobs that control creativity vs...