A Conditional No-Hair Theorem for Transformers: How Language Models Forget Their Prompt↗ (원문, 새 창)

본문 미리보기

arXiv:2610.24133v1 Announce Type: new Abstract: We ask how much an autoregressive transformer retains of its prompt after generating $t$ tokens. Consider two different prompts followed by the same continuation text, so that any difference in their hidden states comes from the prompt alone, and we find that the distance between their hidden states decays approximately as a power law $t^{-\alpha}$, where $\alpha=1$ is the rate at which a fixed prompt's share of uniform attention shrinks and small