In this article, we will focus on checking the outputs generated by the system. Checking outputs before showing them to users can be important for ensuring the quality, relevance, and safety of the responses provided to them or used in automation flows. We will learn how to use the moderation API to check outputs and how to use additional prompts to evaluate the output quality before displaying them.
Let’s dive into the examples. We have already discussed the moderation API in the context of evaluating inputs. Now, let’s revisit it in the context of checking outputs. The moderation API can also be used to filter and moderate outputs generated by the system itself.
Here’s an example: a generated response to the user. We will use the moderation API in the same way as we saw in the earlier video. Let’s see if this output is flagged. As you can see, this output is not flagged and has very low scores in all categories, which makes sense given the response.
In general, it is also important to check the outputs. For example, if you were creating a chatbot for sensitive audiences, you could use lower thresholds for flagging outputs. If the moderation output indicates that the content is flagged, you can take appropriate actions such as responding with a fallback answer or generating a new response.
Note that as we improve the models, they are also becoming less and less likely to return harmful output.
Another approach for checking outputs is to ask the model itself if the generated output was satisfactory and if it follows a certain rubric that you define. This can be done by providing the generated output as part of the input to the model and asking it to rate the quality of the output.
You can do this in various different ways. Let’s see an example. Our system message is ‘You are an assistant that evaluates whether customer service agent responses sufficiently answer customer questions and also validates that all the facts the assistant cites from the product information are correct.’
You could also use a Chain of Thought reasoning prompt for this. This might be a little bit difficult for the model to validate both in one step, so you could play around with this. You could also add some other kind of guidelines, such as asking if the response uses a friendly tone in line with our brand guidelines.
Let’s add our customer message. This is the initial message used to generate this response. Then, let’s add our product information. We fetched this in the previous step for all of the products mentioned in this message.
Now, let’s define our comparison. The customer message is the customer message, this product information, and the agent response. Let’s format this into a messages list and get the response from the model.
The model says yes, the product information is correct, and the question is answered sufficiently well. In general, for these kinds of evaluation tasks, I also think it is better to use a more advanced model like GPT-4.
Let’s try another example. The response is ‘Life is like a box of chocolates.’
The model has determined that this does not sufficiently answer the question or use the retrieved information. This question, ‘Does it use the retrieved information correctly?’ is a good prompt to use if you want to make sure that the model isn’t hallucinating, which is making up things that aren’t true.
As you can see, the model can provide feedback on the quality of a generated output, and you can use this feedback to decide whether to present the output to the user or to generate a new response. You could even experiment with generating multiple model responses per user query and then having the model choose the best one to show the user.
In general, checking outputs using the moderation API is good practice. But while asking the model to evaluate its own output might be useful for immediate feedback to ensure the quality of responses in a very small number of cases, I think it’s probably unnecessary most of the time, especially if you’re using a more advanced model like GPT-4.
I haven’t actually seen many people do something like this in production. It would also increase the latency and cost of your system because you’d have to wait for an additional call for the model, and that’s also additional tokens. If it’s really important for your application that your error rate is 0.0001, then maybe you should try this approach. But overall, I wouldn’t really recommend that you do this in practice.
In the next article, we’re going to put together everything we’ve learned in the evaluate input section, process section, and checking output section to build an end-to-end system.
