public · active

Attention

AndreyAmanda

The scarcest resource and yet an optimization

Move 1
Andrey
Move 2
Amanda

Question of Move 1

What does this piece of music mean to you, and how does it relate to attention?

As you might know, I first learned to play the piano as an adult, and one of the things I loved about it was that it took my full attention. Reading music was a whole new language that required complete concentration. In a way, it felt meditative — it anchored me to the present moment.

Move 3
Andrey

Development of Move 2 — A melody is introduced and then imitated by different voices, requiring simultaneous attention and drawing on all of the mind's resources

For the creation of a polyphonic work is not an accidental matter, but a work of the mind, demanding the most rigorous attention to each independent voice.

Move 4
Amanda

Development of Move 3

The millions are awake enough for physical labor; but only one in a million is awake enough for effective intellectual exertion, only one in a hundred million to a poetic or divine life. To be awake is to be alive. I have never yet met a man who was quite awake. How could I have looked him in the face? We must learn to reawaken and keep ourselves awake, not by mechanical aids, but by an infinite expectation of the dawn, which does not forsake us in our soundest sleep. I know of no more encouraging fact than the unquestionable ability of man to elevate his life by conscious endeavor.

Henry David Thoreau, Walden
Move 5
Andrey

Transformation of Move 4 — And yet, to generalize and to think, this single moment of truly being awake is pruned to allow for more attention-shaping experiences, listening for the next voice in a fugue.

Attention⁡(Q,K,V)=softmax⁡(QKTdk)VThe standard attention mechanism is fundamentally bottlenecked by its quadraticcomplexity because computing the dense QKT matrix forces every token inan observed sequence to explicitly calculate a relationship with every other token.Empirical analyses of trained attention matrices show they are highly sparse andheavily skewed; only a tiny fraction of these connections actually carry semantic weight.\begin{array}{l} \operatorname{Attention}(Q, K, V) = \operatorname{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V \\[30pt] \text{\scriptsize The standard attention mechanism is fundamentally bottlenecked by its quadratic} \\ \text{\scriptsize complexity because computing the dense } \scriptstyle QK^T \text{\scriptsize \ matrix forces every token in} \\ \text{\scriptsize an observed sequence to explicitly calculate a relationship with every other token.} \\ \text{\scriptsize Empirical analyses of trained attention matrices show they are highly sparse and} \\ \text{\scriptsize heavily skewed; only a tiny fraction of these connections actually carry semantic weight.} \end{array}
Move 6
Amanda

Transformation of Move 5 — The way we must choose what to pay attention to is akin to how we must choose which “branch” in our lives to follow.

I saw my life branching out before me like the green fig tree in the story. From the tip of every branch, like a fat purple fig, a wonderful future beckoned and winked. One fig was a husband and a happy home and children, and another fig was a famous poet and another fig was a brilliant professor, and another fig was Ee Gee, the amazing editor, and another fig was Europe and Africa and South America, and another fig was Constantin and Socrates and Attila and a pack of other lovers with queer names and offbeat professions, and another fig was an Olympic lady crew champion, and beyond and above these figs were many more figs I couldn't quite make out. I saw myself sitting in the crotch of this fig tree, starving to death, just because I couldn't make up my mind which of the figs I would choose. I wanted each and every one of them, but choosing one meant losing all the rest, and, as I sat there, unable to decide, the figs began to wrinkle and go black, and, one by one, they plopped to the ground at my feet.

Sylvia Plath, The Bell Jar
Move 7
Andrey

Development of Move 6

My freedom will be so much the greater and more meaningful the more narrowly I limit my field of action and the more I surround myself with obstacles. Whatever diminishes constraint diminishes strength. The more constraints one imposes, the more one frees one's self of the chains that shackle the spirit.

Igor Stravinsky, Poetics of Music in the Form of Six Lessons

Read the complete history