How does an AI make sense of words?

Imagine a game of telephone: a message travels along a line of people, changing a little at each stop. Some earlier language models passed information along a sentence in a similar way. Self-attention gives a word a more direct route to other words it needs. In “the cat sat because it was tired,” connecting “it” with “cat” helps make sense of the sentence. Those connections are learned from examples. The direct route helps information travel; it doesn’t guarantee the AI will get the meaning right.

A shorter route to the words that matter.

You’re waiting at the only open checkout while the other tills sit empty. Now imagine them all opening. That’s the shift in this sketch: the model can work on many word positions together, instead of passing through them in a fixed queue. Computers are good at doing many calculations at once, so this makes training more efficient. There’s a distinction worth keeping: learning from a sentence can happen in parallel, while writing a new reply still happens a piece at a time.

Many positions together when learning; one new piece at a time when replying.

Think of mixing a drink from several bottles. A generous pour from one, a splash from another: the proportions change the result. Attention combines information in a similar way. A query is what a word is looking for; a key helps decide how relevant another word is; its value is the information it contributes. Better matches get a bigger share of the mixture. These are learned number patterns, rather than literal questions or labels. The bottles give us a way to picture what those numbers do.

The proportions make the mixture.

Ask a few friends to read the same sentence and they may notice different things. One spots who did what; another notices what “she” refers to. A Transformer makes several sets of connections at once and combines the results. Each set is called an attention head. The sketch imagines them as a panel of readers. The heads learn their own patterns; nobody assigns them jobs such as “find the subject.” Having several gives the model more ways to connect the same words.

Different connections, brought together.

“The cat sat on the mat” and “the mat sat on the cat” use exactly the same words. The order changes the whole picture. Attention needs help keeping track of that order. The original Transformer adds a position signal to each word’s numerical representation, a little like numbering cards before shuffling them. The signal lets the model use where a word sits as well as what it is. The numbers on the sketch are our shorthand for that extra information.

Same words, different order, different meaning.

Try finishing this sentence: “I put the coffee on the…” You choose what comes next from the words already there. A language model builds a reply in a similar sequence. It predicts a small piece of text, adds it, then predicts the next. These pieces are called tokens; a token can be a word or part of one. During training, a rule called a mask blocks later answer pieces so the model can’t peek ahead. When generating, it has its prompt and the reply so far to work with.

Each new piece joins the context for the next.