public · active · watched by 1

Impermanence of Memory

AndreyLukasz

A biological limitation or an optimization: the human condition.

Move 1
Andrey

Memory is the seamstress, and a capricious one at that. Memory runs her needle in and out, up and down, hither and thither. We know not what comes next, or what follows after.

Virginia Woolf, Orlando
Move 2
Lukasz

Contrast of Move 1 — it contends some memories are set so deeply they feel like reliving the past

I confess I do not believe in time. I like to fold my magic carpet, after use, in such a way as to superimpose one part of the pattern upon another.

Vladimir Nabokov, Speak, Memory
Move 3
Andrey

Synthesis of Move 2 — And yet, whether we command our memories or merely witness them, they are only ever imperfect, partial echoes of the past. We forget what was, to open the space needed to create what will be.

Attention⁡(Q,K,V)=softmax⁡(QKTdk)VThe standard attention mechanism is fundamentally bottlenecked by its quadraticcomplexity because computing the dense QKT matrix forces every token inan observed sequence to explicitly calculate a relationship with every other token.Empirical analyses of trained attention matrices show they are highly sparse andheavily skewed; only a tiny fraction of these connections actually carry semantic weight.\begin{array}{l} \operatorname{Attention}(Q, K, V) = \operatorname{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V \\[30pt] \text{\scriptsize The standard attention mechanism is fundamentally bottlenecked by its quadratic} \\ \text{\scriptsize complexity because computing the dense } \scriptstyle QK^T \text{\scriptsize \ matrix forces every token in} \\ \text{\scriptsize an observed sequence to explicitly calculate a relationship with every other token.} \\ \text{\scriptsize Empirical analyses of trained attention matrices show they are highly sparse and} \\ \text{\scriptsize heavily skewed; only a tiny fraction of these connections actually carry semantic weight.} \end{array}