This article was published as part of the C# Advent 2023, you can check out the rest of the articles here.
Earlier this year Microsoft added a new API to the Azure AI suite called the Content Safety API. This API replaces the Content Moderator API which will be deprecated next year and eventually retired in 2027. As the name suggests, the purpose of this API is to detect harmful content to allow organizations to comply with regulations and / or provide the best experience to their users.
The API provides 4 different types of analysis:
Analyze Text. Scan text for sexual content, violence, hate, and self-harm, the API supports multiple languages and different severity levels.
Analyze Image. Scan images for sexual content, violence, hate, and self-harm, the API supports different severity levels.
Jailbreak risk detection. Scan text for the risk of a jailbreak attack on a LLM.
Protected material text detection. Scans AI-generated text for known text content.
API integration
In this article we will focus on the image and text analysis since they are the only methods available via the C# SDK at the moment, the code samples are from a demo application built using Blazor (you can find the code here). In order to access the API we need to create a content safety resource in the Azure Portal and retrieve the endpoint and keys:
Next, we add the C# SDK which is available as a nuget package:
Once the package is added, we can add a ContentSafetyClient to the application:
builder.Services.AddAzureClients(clientBuilder =>
{
clientBuilder.AddContentSafetyClient(new Uri(builder.Configuration["contentSafetyEndpoint"]),
new Azure.AzureKeyCredential(builder.Configuration["contentSafetyKey"]));
});
The client has 2 methods: AnalyzeImage and AnalyzeText (with corresponding async versions). The AnalyzeImage method takes an AnalyzeImageOption object as a parameter, this object includes the image data:
public async Task<AnalyzeImageResult> AnalyzeImage(Stream imageStream)
{
try
{
var imageData = await BinaryData.FromStreamAsync(imageStream);
ContentSafetyImageData image = new ContentSafetyImageData(imageData);
var request = new AnalyzeImageOptions(image);
Response<AnalyzeImageResult> response;
response = await _contentSafetyClient.AnalyzeImageAsync(request);
return response;
}
catch (RequestFailedException ex)
{
Console.WriteLine("Analyze image failed.\nStatus code: {0}, Error code: {1}, Error message: {2}", ex.Status, ex.ErrorCode, ex.Message);
throw;
}
}
The AnalyzeImageResult object returned by the method includes the severity per category, which we can then display in our application:
Below are the results when running a test with an image from a "Fatality" from the Mortal Kombat videogame:
The AnalyzeText method takes an AnalyzeTextOptions object as a parameter, which allows you to specify the text and the output type. Changing the output type updates the severity scale so you can filter at a more granular level. In addition to that, you can specify a "blocklist" which is a list of terms that might not be identified by the base model or that are specific to your use case.
public async Task<AnalyzeTextResult> AnalyzeText(string text, int outputType)
{
try
{
var request = new AnalyzeTextOptions(text);
request.BlocklistNames.Add("DemoBlocklist");
request.OutputType = outputType == 1 ? AnalyzeTextOutputType.FourSeverityLevels :
AnalyzeTextOutputType.EightSeverityLevels;
Response<AnalyzeTextResult> response = await _contentSafetyClient.AnalyzeTextAsync(
request);
return response;
}
catch (RequestFailedException ex)
{
Console.WriteLine("Analyze image failed.\nStatus code: {0}, Error code: {1}, Error message: {2}", ex.Status, ex.ErrorCode, ex.Message);
throw;
}
}
On the application side, we can test this with any text we want. Below are the results when analyzing text with different output types:
The content safety resource used during testing is using a blocklist with the term "developer" in it. This is an example of the result:
As you can see, the severity and blocklist results are independent of each other, so even if the text is not flagged in any of the 4 existing categories it will still be flagged if it contains any of the terms in the blocklist.
Content safety studio.
If you don't want to create a POC application to evaluate the API's capabilities Microsoft has you covered! Content Safety Studio is an online tool that you can use to do that and more, additional functionality includes:
Create and edit blocklists.
Test preview functionality.
Monitor activity (image below).
Summary.
The Azure Content Safety API is a great option to easily identify harmful content in text and images across 4 categories: sexual content, violence, hate, and self-harm. The results are also scored according to the severity of the content so the appropriate actions can be taken. Additional functionality currently in preview allows for content filtering in LLM models and is available through the Content Safety Studio tool.
Additional resources:
Top comments (0)