Hello, welcome to this short video. My name is Professor Phil Newton, and in this video, we’re going to talk about the risks that ChatGPT and similar tools currently pose to the way that we assess students in higher education. We’ll also discuss some ways in which we can mitigate those risks.
The framework we’re going to use for this presentation is a tool that we’ve developed called Tilapia - the Tool for Identifying Learning Assessments Potentially Impacted by Artificial Intelligence. Tilapia allows us to identify the features of assessments that make them more or less likely at risk from the integrity challenges of ChatGPT.
Before we dive into the details, let’s clarify that assessment security is just one feature of assessment design. This framework won’t determine whether an assessment is good or bad, valid or invalid, reliable or not, or inclusive or not. It will simply tell us whether an assessment is more or less secure in the face of the challenges posed by ChatGPT.
Now, let’s take a closer look at ChatGPT itself. ChatGPT is a sophisticated chatbot that has received a lot of attention regarding its ability to pass assessments. There are three main narratives that have emerged: one claims that ChatGPT can pass all assessments and poses an existential crisis, another argues that ChatGPT isn’t very good and makes lots of mistakes, and the third sees ChatGPT as a disruptor that should be embraced as a huge opportunity.
In reality, there are different versions of ChatGPT, and its capabilities are evolving rapidly. Currently, there are two versions: a free one and a subscription one. The subscription version performs much better, scoring over 75% on average in exams, while the free version scores just over 50%. This means that the subscription version can enable someone to pass difficult qualifying exams in fields like medicine and law, despite having no prior knowledge.
However, it’s important to note that foundational knowledge is crucial for students to know what to ask ChatGPT and how to apply its responses. Foundational knowledge is the basis for higher-level cognition and critical thinking. While ChatGPT can assist in practical tasks and problem-solving, it still relies on core knowledge.
Now, let’s discuss the features of assessments that make them more or less secure in the context of ChatGPT. These features include the specificity of the assessment, the level of Bloom’s taxonomy it targets, whether it’s a practical or written assessment, whether it’s conducted in-person or online, whether the questions are seen or unseen, the level of supervision or invigilation, whether it’s a real-time or asynchronous assessment, and whether it’s open book or closed book.
Each of these features interacts and overlaps with others, and none of them is perfect. Assessments that are more specific, target foundational knowledge, are practical, conducted in-person, have unseen questions, and involve supervision are generally more secure. On the other hand, assessments that are general, target advanced learning, are written, conducted online, have seen questions, lack supervision, are asynchronous, and are open book are more at risk from ChatGPT.
It’s important to note that no assessment design can completely guarantee against cheating or fraud. However, by considering these features and designing assessments that prioritize foundational knowledge and authenticity, we can make assessments more secure. It’s also crucial to keep in mind the inclusivity of assessments and the diverse needs of students.
In conclusion, the risks posed by ChatGPT to assessment integrity require us to reconsider assessment design. While ChatGPT can be a powerful tool, it should not replace the need for students to demonstrate their learning in a reliable, valid, and meaningful way. The Tilapia tool can help identify assessments at risk and guide the redesign process. It’s important to strike a balance between security and the purpose of assessments.
If you’re interested in learning more about this topic, I’ve provided references in the video description. Feel free to reach out to me via email if you have any questions. Thank you for watching, and I’ll see you in another video. Bye-bye.
