Because we have seen a couple of papers in this line already, including from OpenAI, I did not start off with that as the original part. It is still pretty original, though. People are training models based on the individual steps of reasoning and not just on the final result. What they are hinting at in this paper, using synthetic data, does not necessarily limit you to the same capability as the model that generated the synthetic data. People who own interesting datasets do not have quite the lock or the stranglehold that maybe we thought they did. Earlier this year, Microsoft invested 10 billion dollars into OpenAI. This massive investment allows Microsoft to make a huge amount of profit. They have the right to 75 percent of OpenAI’s profits until their initial investment is returned, and then they can take 49 percent of OpenAI’s profits until they receive 92 billion dollars in profit from the deal. But interestingly, Microsoft is making it harder for OpenAI to generate income by adding their own competition into the mix. And weirdly, they are giving it away for free. Now, what I mean by this is that the AI world is split into two camps. On the one hand, you have the community-built open source models, and on the other, the privately owned models. While the open source models rely on community effort and very little by way of funding, the case is different for privately owned models. Privately owned models are within the vaults of Silicon Valley big boys like OpenAI, Google, and Meta, who boast superior money power. Predictably, open source models come into the mix with the romantic idea of beating the Silicon Valley behemoths at their game. But time and again, the privately owned models with their billion-dollar deep pockets have always proven to be the kings of the game. At least, this has been the case until Microsoft’s latest model came into the picture. You see, recently, Microsoft unexpectedly burst onto the scene with Orca, making it the first time a private behemoth was delving into the realms of open source community-built AI models. For clarity’s sake, Orca is an open source AI model, much smaller than large language models like ChatGPT. However, in this fight, size doesn’t seem to matter because Orca comes with some great features, such as the innovative training model. It is with this innovative training model that Microsoft’s newest innovation is taking on the Silicon Valley giants in their own game. Not to mention that Orca has completely swept out the longest-standing king of open source AI models, Vikuna. But what is Orca, and why is it so special? And most importantly, what is Orca bringing to the table that will benefit AI users? You see, the AI game is a money game, and only the rich can play. Because we are talking about models with billions of parameters in them that cost millions to gather. This involves very costly processes, such as gathering vast and adequate data, training-based models, fine-tuning them, and reinforced learning from human feedback. For anyone that has used Amazon Web Services, you know how easy it is for costs to spiral out of control. And this causes a major barrier to entry for smaller competitors. However, there is a lifeline enabling these smaller firms to play the game and even get ahead. It’s called distillation, which essentially allows the smaller players to learn from the bigger ones, like OpenAI. Distillation essentially involves tapping into the response from larger models, like the GPT-4, to teach smaller models to imitate them. And the thinking behind this move is, once we have the large models, we could also have much smaller models to replicate some of their features by simply imitating them while maintaining their size. So, while the large language models or LLMs do the major work of learning all they can about our world, the smaller models learn from this vast reservoir of knowledge to replicate their actions without doing the heavy work. A great example of working smarter and not harder. And this is where Microsoft Orca comes in. So, basically, Orca is a 13 billion parameter model specially fit to blow up the capacity of smaller models through distillation. It actually latches onto GPT-4, learns from its rich and vast signals, which include explanation traces, step-by-step thought processes, and other really high-level instructions. This super smart student understudies the master and learns everything possible from it by absorbing its vast knowledge and really understanding the reason behind every one of its decisions. Orca takes things a notch higher with its progressive learning capabilities. But how does it achieve this? Orca takes in large-scale and diverse imitation data that helps it constantly learn and adapt. This actually makes it a continuously evolving tool that is as powerful, or maybe more powerful, when you stack it side by side with the master.
