babel

Research + Learning

Level With Me

Put new research to work without needing a PhD in the subject. Level With Me makes difficult papers understandable, with worked examples that show how the ideas apply to your own project.

Download INKO

How does an AI figure out what “it” means?

What I already knowI use AI, but I do not know how it works.

Paper: Attention Is All You Need
Vaswani and colleagues · 2017

Read these two sentences:

The trophy would not fit in the suitcase because it was too big.

“It” means the trophy. The trophy is too big to fit.

The trophy would not fit in the suitcase because it was too small.

Now “it” means the suitcase. The suitcase is too small to hold it.

The word “it” has not changed. Its meaning comes from its relationship with the other words. That is the problem attention helps a language model handle.

Let the words inform each other

A model represents words, or pieces of words, as lists of numbers. An attention step compares these representations, assigns different weights to them, and combines information from them. A word’s representation can then reflect the sentence around it.

For “it,” the useful connections include the trophy, the suitcase and the clue at the end. The comparisons are learned during training. Nobody has to write a separate rule for every possible sentence. This is called self-attention because the words are drawing information from the same text. The paper, §3.2.

Why this was a breakthrough

Many earlier models processed a sentence in sequence, passing information from one step to the next. A connection between distant words had to travel through those steps. The Transformer let words draw information directly from other positions. Much of that work could happen at the same time, making training a better fit for powerful parallel computers. The researchers demonstrated better translation results with less training computation. Co-author Jakob Uszkoreit’s explanation.

Word order still matters. “The dog chased the cat” and “The cat chased the dog” contain the same words. The Transformer adds information about position so it can distinguish them. And although training can process many positions together, the original model writes its translation one piece at a time. The paper, §§3.1 and 3.5.

Several relationships at once

The paper also uses multi-head attention: several sets of these learned comparisons running alongside one another. They let the model combine different relationships instead of relying on one mixture of information. Each head repeats the same basic operation: a word gets its meaning in context by drawing on the words around it. §3.2.2.

Make a research paper your secret weapon.

In March 2016, a computer played a move that professional Go commentators thought was a mistake. It helped win the game. AlphaGo went on to beat champion Lee Sedol four games to one. Demis Hassabis, co-founder of the AI lab DeepMind, saw a reason to pursue his larger ambition: use machines to find answers to scientific problems that people had struggled with for decades. One of them was understanding the shapes of proteins. [1]

  1. 2016 · AlphaGo

    Take the idea beyond the game.

    “the technology was ready to be applied to our real goal of accelerating scientific breakthroughs.”

    Demis Hassabis, recalling what AlphaGo’s victory meant to him [1]

    The team’s 2016 paper explained how AlphaGo learned from expert games, improved by playing against itself, and looked ahead to compare promising moves. It learned which possibilities deserved attention in a game with far too many positions to examine them all. David Silver and his colleagues had combined learning with search to make a problem of that size manageable. [2]

    Hassabis believed the success justified attempting a much harder scientific problem. A protein starts as a chain of chemical building blocks. Its folded shape helps determine what it does inside a cell and what can attach to it, including a drug. Scientists could read the chain’s sequence, yet spend years working out its shape. AlphaFold would need biological evidence and a new design to predict that shape. [1][3]

  2. 2017 · Attention Is All You Need

    Borrow an idea from translation.

    A separate team at Google was working on translating languages. In their 2017 paper, Ashish Vaswani and his colleagues introduced the Transformer. Its central operation let each part of a sentence draw information from other parts, including words far away. The model could learn which relationships mattered. [4]

    Consider “she sat on the bank and watched the ducks.” You use the rest of the sentence to understand “bank.” Attention gives a machine a way to combine information across the sentence too: it compares representations of the words, assigns weights to them, and uses those weights to decide how much information to take from each. The paper also showed how to do much of this work in parallel, speeding up training. [4][5]

    That idea had uses beyond language. In a folded protein, two parts far apart along the chain can sit beside each other. Predicting its shape requires understanding relationships between parts. The AlphaFold team adapted attention to that problem, combining it with knowledge of protein sequences and three-dimensional geometry. [6]

  3. 2020 · AlphaFold 2

    Work out which parts belong together.

    The team, led by John Jumper, trained AlphaFold using protein shapes measured in laboratories and information from related protein sequences. The system repeatedly exchanged clues about the chain with clues about how its parts might fit together, then refined its predicted shape. Here, the learning depended on decades of experimental biology. [6]

    In a blind assessment in 2020, AlphaFold’s predictions approached experimental accuracy for many test proteins. The team published its methods in 2021, including the attention-based design. A research idea developed for translation had contributed to a tool that biologists could use. [3][6]

  4. 2022 · A use for the discovery

    Find a way to interrupt malaria.

    At Oxford, Matthew Higgins’s team had spent years trying to see the full shape of a protein on the malaria parasite. They wanted to develop a vaccine with an unusual effect: when a mosquito bit a vaccinated person, it would swallow antibodies that could interfere with the parasite inside the mosquito, stopping it from reaching the next person. They needed to know where those antibodies could attach. [7]

    Combining AlphaFold’s prediction with their experimental data helped the team complete the protein’s structure. Together with antibody experiments, that gave them a basis for designing a vaccine around the whole protein, including useful targets beyond the fragment many earlier designs used. They had a clearer route for vaccine development; protection in people remained to be tested. [8]

Hassabis pursued biology after seeing what his team could accomplish in Go. Jumper’s team drew on research from another field to help make that ambition practical. Level With Me gives you a way into knowledge that could change what you can attempt. Start with something you want to understand or do, and what you already know. It explains a paper through examples you can follow, then connects the finding to your question. You can learn enough to use an idea from an unfamiliar field without first earning a degree in it.