Improving Task Evaluation in Natural Language Processing

In this article, we will focus on tasks to evaluate inputs, which are important for ensuring the quality and safety of the system. For tasks that require multiple sets of instructions to handle different cases, it can be beneficial to first classify the type of query and then use that classification to determine which instructions to use. This can be achieved by defining fixed categories and hard-coding instructions that are relevant for handling tasks in a given category. For example, when building a customer service assistant, it might be important to first classify the type of query and then determine which instructions to use based on that classification. Let’s consider an example to make this clearer. Suppose we have a system message that instructs the model to classify customer service queries into primary and secondary categories and provide the output in a JSON format. The primary categories can be billing, technical support, account management, or general inquiry. The secondary categories can be unsubscribe, upgrade, and so on.

Now, let's take a user message as an example: 'Delete my profile and all of my user data.' Based on this message, we can classify it as account management and close account as the secondary category. By asking for a structured output like JSON, we can easily read this into an object and use it as input for subsequent steps. Another example user message could be 'Tell me more about your flat screen TVs.' In this case, the model correctly categorizes it as a general inquiry about flat screen TVs. Based on the categorization of the customer inquiry, we can provide more specific instructions to handle the next steps. For example, we can provide additional information about the TVs or give a link to closing the account, depending on the category. In the next video, we will explore more ways to evaluate inputs and ensure responsible usage of the system.