In this lesson, we will cover what a Transformer is and how it allows us to take in multiple input words and generate multiple output words. So far, we have explained how we can convert one French word into one English word. Now, our goal is to convert multiple French words into multiple English words. To achieve this, we will modify our approach. Instead of translating from one word to another, we will translate from the previous word and predict the next word. This change in approach is represented in a diagram, where our model takes in the French phrase and the previous word as inputs and predicts the next word. We refer to the French phrase as the prompt and distinguish it from the previous word. To generate the first word, we pass in the prompt and a magical start word. This start word represents the beginning of the sequence. To generate the next word, we feed in both the prompt and the previous word. We continue this process iteratively until we reach the end of the sequence word. This process is known as auto-regressive decoding, where we predict the next word one at a time. The entire pipeline can be summarized as follows: we convert the prompt and the start of the sequence into the first word, and then predict the next word based on the previous tokens. We incorporate context by taking a weighted sum of the vectors representing the prompt and the previous words. This weighted sum is then fed into a multi-layer perceptron to perform computation. The computation produces a vector, which is then mapped to the nearest word. This pipeline allows us to convert multiple input words into multiple output words. We have introduced two new concepts in this lesson: auto-regressive decoding and context incorporation. Auto-regressive decoding predicts the next word one at a time, while context incorporation involves incorporating information from the prompt into the previous words. The pipeline diagram shows how these concepts are applied. However, we have not discussed in detail how the weighted sum for context incorporation is computed. This is known as self-attention, which will be covered in a future lesson. In conclusion, we have provided an overview of the Transformer and its multi-word pipeline. While we have simplified the architecture, this serves as a starting point for understanding the key ideas behind language models. For more resources and slides, please visit the course website.
