OpenAI’s GPT-3 language model has gained immense popularity worldwide.
If you are not familiar, this model serves as the foundation for the viral sensation known as ChatGPT. If you encounter coding difficulties, ChatGPT can assist you. Simply explain the problem statement, and ChatGPT will generate a code snippet for you. However, ChatGPT offers more than just coding assistance. It can also create short bedtime stories for children using simple language and easy-to-follow plots. Moreover, ChatGPT has earned the nickname ‘Google Killer’ due to its exceptional ability to explain complex concepts. You can ask ChatGPT to explain challenging topics like Quantum Computing, and it will clear any doubts you may have. Whether you require coding assistance or want to explore various subjects, ChatGPT is here to help!
However, every innovation has its drawbacks, and ChatGPT is no exception. The primary drawback is the cost associated with developing and maintaining ChatGPT, which can be substantial. Analysts and technology experts estimate that training a large language model like GPT-3 could cost over $4 million or ₹32,83,44,000 (32 crore rupees). With that amount of money, you could purchase a luxurious 6BHK sea-facing flat in Marine Lines, India, and still have enough left to buy 10 Land Rover Defender 130X.
To comprehend the significant cost of $4 million or 32 crore rupees and the extensive infrastructure required to train and maintain a model like ChatGPT, it is crucial to understand what existed before ChatGPT. Prior to OpenAI’s ChatGPT gaining global recognition for its impressive sentence generation, there was a small startup called Latitude that amazed users with its AI Dungeon game. This game allowed people to use AI to craft imaginative stories based on their prompts. However, as AI Dungeon grew in popularity, Latitude’s CEO Nick Walton noticed a rapid increase in expenses for maintaining the text-based role-playing game. AI Dungeon relied on text-generation software powered by GPT language technology, which was provided by OpenAI, a Microsoft-supported AI research lab. The more people played AI Dungeon, the higher the bill that Latitude had to pay to OpenAI. During its peak in 2021, Latitude had to spend approximately $200,000 or ₹1,64,14,940 (1 crore rupees) per month on OpenAI’s generative AI software and Amazon Web Services. This was necessary to handle the large number of user queries they received daily. The high costs faced by Latitude due to their AI bills highlight an important reality about the recent surge in generative AI technologies. Developing and keeping up with this software can be extremely expensive, not only for the companies creating the underlying technologies (known as large language or foundation models), but also for those using AI to enhance their own software. In simpler terms, it can be quite pricey to create and maintain these advanced AI systems, whether you’re the company behind them or the one using them in your software. Large companies like Microsoft, Meta (formerly Facebook), and Google leverage their significant resources to establish a strong position in technology that smaller competitors find challenging to match.
Returning to ChatGPT, training GPT-3 comes with a hefty price tag of over 4.6 million dollars. It utilizes a Tesla V100 cloud instance and takes up to 9 days of training time. The cost of training and running large language models is significantly higher compared to previous computing advancements. Even after building or training the software, running these models requires extensive computing power because they perform billions of calculations for each response they generate. In contrast, serving web apps or pages involves much less computation. Furthermore, these calculations necessitate specialized hardware. While traditional computer processors can handle machine learning models, they are slower in comparison. Most training and inference processes now rely on graphics processors, known as GPUs, which were originally designed for 3D gaming. However, they have become the standard for AI applications due to their ability to perform numerous simple calculations simultaneously. Nvidia is a leading provider of GPUs (Graphics Processing Units) for the AI industry. Scientists and researchers who develop these models often humorously refer to the intense computational demands of their work, joking that they can ‘melt GPUs.’ According to analysts and technologists, training a large language model like OpenAI’s GPT-3 could cost more than $4 million. More advanced models can even reach costs in the billions. For example, Meta’s recent LLaMA model used 2,048 Nvidia A100 GPUs and took around 21 days to train on 1.4 trillion tokens. This training process consumed approximately 1 million or 10 lakh GPU hours, which, with dedicated prices from AWS, would amount to over $2.4 million or Rs. 19,69,69,920 (19 crore rupees). Despite being smaller than OpenAI’s current GPT models like ChatGPT-3, which has 175 billion parameters, Meta’s model required significant computational resources. Clement Delangue, CEO of AI startup Hugging Face, mentioned that training their Bloom large language model took over two and a half months and needed access to a supercomputer equivalent to about 500 GPUs.
Now, let’s discuss the cost of inference. When using a trained machine learning model to make predictions or generate text, there is a process called ‘inference.’ Surprisingly, this process can be more expensive than training because it may need to run millions of times, especially for popular products. For instance, a widely-used product like ChatGPT, which is estimated to have had 100 million monthly active users in January, could have cost OpenAI around $40 million or ₹3,280,082,280 (328 crore rupees) just to process the numerous prompts people entered into the software during that month. The costs can rise even higher when these tools are used billions of times each day. Financial analysts estimate that Microsoft’s Bing AI chatbot, powered by an OpenAI ChatGPT model, requires infrastructure investment of at least $4 billion or ₹328,096,520,000 (32809 crore) to handle all the responses for Bing users. Currently, people rely on NVIDIA GPUs for most of their inference tasks. These GPUs are quite expensive, especially the DGX systems. However, there is a challenge when it comes to inference workloads. If the workload suddenly increases rapidly, like what happened with ChatGPT reaching a million users in just five days, the GPU capacity can’t keep up. GPUs are primarily designed for training and graphics acceleration, not for handling such spikes in user demand. GPUs are expensive, power-hungry, and their architecture is optimized for parallel processing, which is more beneficial during the training phase. Several startups are working on creating specialized hardware to improve the efficiency of inferencing tasks. Some notable examples include Graphcore, which develops AI processors called Intelligence Processing Units (IPUs) that speed up both training and inferencing. Habana Labs focuses on developing inference processors that offer high performance and energy efficiency for data centers and cloud environments. Cerebras Systems is known for its large-scale processors designed for AI workloads, including inferencing. Mythic is developing AI chips that aim to deliver high performance with low power consumption, specifically for inferencing. These startups, among others, are dedicated to creating hardware solutions that optimize and accelerate inferencing tasks, addressing the limitations of GPUs in this area.
In recent news, two AI chip startups, Neuchips and SiMa, have outperformed Nvidia GPUs in the latest MLPerf AI inference benchmarks. These startups showcased superior performance per watt in data center recommendation and edge image classification tasks, surpassing Nvidia’s H100 and Jetson AGX Orin scores, respectively. Additionally, Qualcomm’s Cloud AI100 demonstrated favorable power efficiency compared to Nvidia H100 in various metrics. These results indicate that these AI chip startups are making strides in delivering efficient and powerful solutions for AI inference tasks.
