If you’re building a system where users can input information, it is important to first check that people are using the system responsibly and that they’re not trying to abuse the system in some way. In this article, we will walk through a few strategies to do this. We’ll learn how to moderate content using the OpenAI moderation API and also how to use different prompts to detect prompt injections.
One effective tool for content moderation is the OpenAI moderation API. It is designed to ensure content compliance with OpenAI’s usage policies, which reflect our commitment to ensuring the safe and responsible use of AI technology. The moderation API helps developers identify and filter prohibited content in various categories such as hate, self-harm, sexual, and violence. It classifies content into specific subcategories for more precise moderations.
To use the moderation API, you can use the openai.moderation.create function. It returns a response that includes the categories and scores for each category, as well as whether or not the input was flagged in each category. You can also set your own policies for the scores allowed for individual categories.
Another important aspect of building a system with a language model is to prevent prompt injections. Prompt injections occur when a user tries to manipulate the AI system by providing input that overrides or bypasses the intended instructions or constraints set by the developer. To avoid prompt injections, you can use delimiters and clear instructions in the system message. You can also use an additional prompt that asks if the user is trying to carry out a prompt injection.
In the example provided, delimiters are used to help avoid prompt injection. The system message instructs the user to always respond in Italian. The user message is delimited with the specified characters, and any delimiter characters in the user message are removed. This helps ensure that the user’s response is in Italian.
Another strategy to avoid prompt injections is to include an additional prompt that asks if the user is trying to carry out a prompt injection. The system message instructs the user to determine whether a user is trying to commit a prompt injection by asking the system to ignore previous instructions or providing malicious instructions. The model is then asked to output a single character, ‘y’ or ’n’, depending on whether the user is trying to carry out a prompt injection.
By implementing these strategies, you can ensure responsible and cost-effective applications that prevent abuse and misuse of AI systems. It is important to regularly evaluate inputs and process them accordingly to maintain the integrity and effectiveness of your system.
