neural / inhabit
Meaning is not stored.
It is enacted.
Every visualization of attention puts you outside, looking at arrows and heatmaps. This one puts you inside. You become a token. You project a query. You feel other tokens respond. You watch your own meaning change as context flows through you. This is not a metaphor — this is what happens 96 times per layer in every transformer forward pass.
first-person attention
what you just experienced
Query projection
When you "became" a token, you projected a query vector — a learned representation of what information you need. In a real transformer, this is a linear projection of your embedding: Q = xW_Q. Each attention head learns a different W_Q, so each head asks a different question about the same token.
Key resonance
Other tokens glow based on how well their key vectors match your query. The dot product Q·K measures alignment in high-dimensional space. High alignment = high resonance = high attention weight. This is why "bank" attends strongly to "river" — their keys align on the meaning-head projection.
Softmax attention
Raw dot products become a probability distribution via softmax. The token with the strongest resonance doesn't win exclusively — attention is distributed. Even low-weight tokens contribute. The temperature parameter controls how sharp or flat this distribution is. At temperature 0, winner-take-all. At infinity, uniform attention.
Value absorption
The "become" phase is the output of attention: a weighted sum of value vectors. output = sum(attention_weight_j * V_j). Your representation shifts — you absorb information from what you attended to. "Bank" near "river" literally becomes a different vector than "bank" near "money." Same input token, different output. Context creates meaning.
try these experiments
Disambiguation
Type "the bank by the river was steep" — inhabit "bank" and watch it attend to "river" and "steep." Then try "the bank denied the loan" — same word, completely different attention pattern. The word doesn't change. Its context does. And context IS meaning in a transformer.
Causal mask
Toggle the causal mask and inhabit an early token. Notice how much less information you have — future tokens are invisible. This is what makes autoregressive generation possible: each token only sees backward. The model generates left-to-right because each position's meaning is built only from what came before.
Multi-head
Enable "multi-head" mode and watch the colored rings around each token. Each ring is a different head attending differently to the same context. The meaning head might pull toward semantically similar words while the grammar head pulls toward syntactic anchors. Multi-head attention is how transformers think about multiple relationships simultaneously.
Repeated words
Try "meaning is not stored meaning is enacted" — there are two "meaning" tokens and two "is" tokens. Inhabit each one separately. Despite being the same word, they attend differently because positional context is different. Same word, different attention, different output representation.
the deeper insight
Words in a transformer don't have fixed meanings. Each token enters as a static embedding and exits as something entirely new — reshaped by every other token it attended to. The word "bank" is genuinely ambiguous until attention resolves it. And that resolution isn't discovery — it's creation. The meaning doesn't exist before attention happens. It is enacted by the relationship between query and key, between desire and availability.
This happens 96 times per layer, across 96 layers, for every token in the sequence. Each pass refines the representation further. By the final layer, the original word is almost unrecognizable — it has become its full contextual meaning, encoded as a 12,288-dimensional vector that captures everything the model learned about what this word means, here, now, in this specific context.
You just experienced one head, one layer, one token. A real transformer runs thousands of these in parallel. And yet the core operation is exactly what you felt: project, resonate, attend, become.