what does the unembedding layer do to intermediate activations?

if you embed a token and immediately unembed it, do you get the token back? the answer is yes in "tied" models such as GPT-2, but no in "untied" models (Qwen2.5-72B). what happens between the embedding and unembedding layer - do the activations live in the same space?

words go in fresh (no context); check out "antidisestablishmentarianism".

how to read this page

controls: pick a model, a word, and — for multi-token words — which token’s trajectory to look at. tied models share one weight matrix between embedding and unembedding; untied models learn them separately.
unembedding: “logit lens” applies the model’s final norm before the unembedding head (the standard convention); “raw” applies the bare head. the “output” row is always the model’s own head output, its real prediction.
token display: “raw tokens” shows tokenizer-internal spelling (Ġ = leading space); “decoded” shows the actual string (␣ = space, ␊ = newline).
each row = one depth. its chips are the 5 most likely tokens if you unembed right there — darker fill = more probable, the small number is the log-probability. a solid ring marks the input token itself; a dashed ring marks the word’s actual next token. hover any chip for both spellings and the exact probability.
badges: self #N = where the input token ranks in that depth’s distribution (#1 = perfect round-trip). next #N = where the word’s true next token ranks.
chart: the same story across all depths — rank of the input token (solid) and of the model’s eventual top-1 prediction (dashed), log scale, rank 1 at the top.
⋯ folded rows: stretches of layers where nothing moves (same top-1, self and final-top-1 ranks changing by less than half an order of magnitude) are faded out; click to expand them.