Thanks to everybody for joining today. In this article, I will be discussing my experience applying ChatGPT4 as a static code analysis tool. I will provide examples of how I am using this tool to help review my programming assignments at Vanderbilt University. However, the technique I am describing can be applied to many other contexts, such as reviewing code for partners, sponsors, customers, or even code that we write ourselves. Additionally, this technique can also be used to analyze documents.
This particular use case falls within the scope of the Software Engineering Institute’s National Software Agenda study. Many of you may be familiar with this study, especially the part that focuses on the use of AI-augmented software development tools and methods.
To give you a broader view of how I am using ChatGPT4, I have uploaded videos to my YouTube channel. These videos cover a course I taught at Vanderbilt, where students built a movie recommendation web app using both AI and non-AI parts. The goal of using generative AI in my teaching and research is to make the process more creative, fun, and effective.
The barrier to entry for applying generative AI is low. You just need to create an account on platforms like OpenAI, Bard, Google, or Cloud. While the out-of-the-box capabilities are somewhat limited, you can access more powerful tools by subscribing to a monthly plan. These tools can greatly enhance your programming experience.
However, it is important to note that generative AI tools like ChatGPT4 are not flawless. They have their limitations, and it is crucial to adjust our expectations accordingly. Some computer scientists have made bold statements about the limitations of AI, but I believe these critiques are misplaced. The key is to use the tools properly, think about how people actually apply them, and choose the right tools for the job.
In my case, I have been using ChatGPT4 to help me review and revise programming assignments. I have discovered that these tools can generate very accurate solutions to assignments if the specifications are detailed. To address this, I have started generalizing the specifications to make them harder to reverse engineer. This approach not only challenges the students to be more creative but also streamlines the reviewing process.
To improve the reviewing process, I have trained ChatGPT4 to analyze student submissions for frequently made mistakes. I have created a frequently made mistakes file, which I use to generate natural language queries for ChatGPT4. This allows my graders to provide better feedback and interpret the rubric more precisely. It automates many steps in the reviewing process and helps avoid inter-rater reliability issues.
I have also discovered that large language models like ChatGPT4 can iteratively improve themselves. By engaging in a dialogue with the model and refining the queries, I have been able to enhance its accuracy and effectiveness.
In conclusion, using ChatGPT4 as a static code analysis tool has greatly benefited my teaching and research. It has made the reviewing process more efficient, improved feedback for students, and allowed for more creative assignments. While there are limitations to generative AI tools, when used properly, they can be powerful aids in programming and software development.
References:
