Hello everyone! In this video tutorial, we will learn about how ChatGPT was trained. But before we dive into that, let me clarify a few things. ChatGPT is an application that internally makes a call to OpenAI API. OpenAI API, in turn, uses GPT 3.5 and GPT 4 models to generate a response. So, when a user asks a question, the ChatGPT application breaks it down into smaller parts and sends them to OpenAI API, which generates a response using the GPT models. If you have a premium subscription, you can use the more advanced GPT 4 model; otherwise, GPT 3.5 is used.
Now, let’s say you want to create a book summary generator application. In this application, the user simply inputs the name of a book, and the application generates a summary of that book. The book summary generator is similar to ChatGPT, but it focuses specifically on generating book summaries. The user’s input prompt is passed to OpenAI API, which uses either GPT 3.5 or GPT 4 model to generate the summary. The generated summary is then displayed in the book summary generator application.
Now, let’s take a look at how ChatGPT is trained. The training process can be divided into three stages. In the first stage, generative pre-training is performed. This involves training the base GPT model on a large amount of text data, including books, articles, and blogs from the internet. The GPT 3.5 model, for example, has been trained with 175 billion parameters. The output of this stage is the base GPT model.
In the second stage, supervised fine-tuning is performed. This involves training the base GPT model on a specific task using a dataset of request-response pairs. The dataset is created by collecting real conversations where a human agent acts as a chatbot and responds to user requests. The responses are ranked by the human agent, and a reward model is created based on the rankings. Proximal policy optimization is then applied to update the reward model and improve the responses of the ChatGPT model.
In the third stage, reinforcement learning through human feedback is implemented. This further improves the accuracy of the ChatGPT model. The human agent sends a request, and the ChatGPT model generates a response. The responses are ranked by the human agent, and a reward model is created. Proximal policy optimization is used to update the reward model and improve the responses.
To summarize, ChatGPT is trained through generative pre-training, supervised fine-tuning, and reinforcement learning. The training process involves training the base GPT model on a large amount of text data, fine-tuning it on specific tasks using request-response pairs, and further improving it through reinforcement learning. This process ensures that ChatGPT can generate accurate and contextually appropriate responses.
Thank you for watching this tutorial!
