In this article, we will discuss the changing behavior of ChatGPT over time. Researchers at Stanford and UC Berkeley collaborated to evaluate the performance of GPT 3.5 and GPT 4, two widely used large language models. The evaluation was conducted in March 2023 and June 2023, focusing on diverse tasks such as solving math problems, answering sensitive questions, generating code, and visual reasoning.
The findings revealed that GPT 4’s performance had significantly dropped, while GPT 3.5 had shown some improvement. However, due to the closed-source nature of GPT, detailed information about the models is not publicly available.
Let’s consider a scenario where a user named Vinod is using a large language model for jokes. Initially, Vinod is not satisfied with the model’s responses, but after multiple queries, he starts to enjoy the jokes. The model stores the data and labels it based on human feedback. After a synchronization interval, the model undergoes retraining or fine-tuning.
There are several factors that may contribute to the degradation in performance. One possible reason is the sync interval. If the interval is too long, the model may become out of distribution, resulting in hallucinations. Additionally, tweaking the batch size and architectural issues can also affect the model’s performance.
It is crucial to monitor the stability and drift of large language models. For instance, if a chatbot based on GPT is used in the medical domain to assess mental health, a significant drop in performance could have severe consequences for the business. Continuous evaluation of llm’s quality is necessary for downstream tasks.
The evaluation results showed that GPT 4 had a significant drop in performance compared to GPT 3.5. The verbosity of GPT 4 decreased, while GPT 3.5 showed an increase. Safety checks may have been introduced in GPT 4 to block illegal questions, resulting in fewer sensitive question responses. The overlap between past and present outputs was high in GPT 3.5, indicating consistency.
In conclusion, large language models like GPT are not easily interpretable, making it challenging to identify performance issues. Monitoring the quality of these models is crucial, especially for business applications. A framework for periodic evaluation and measures to improve model quality should be developed. Open-source models like llma2 may offer more control and become a viable alternative to closed-source models like GPT.
Thank you for reading. Have a great weekend!
