DEV Community

Shubham Anandani
Shubham Anandani

Posted on Originally published at imshubh.com on

Content filters for prompts and outputs

Problem

GenAI agents must resist prompt injection, avoid harmful outputs, and maintain provenance. Azure AI Content Safety offers moderation for text/images and safety features (Prompt Shields, groundedness detection, protected material) with configurable thresholds and blocklists.

Options and trade-offs

  • Prompt Shields (direct and indirect injection): Detect jailbreak attempts and document-borne attacks; low false-positive rates; add latency; configurable block vs annotate modes.
  • Groundedness detection (Preview): Detect ungrounded completions relative to sources; helps provenance and QA; preview scope limits.
  • Protected material detection (text GA, code preview): Identifies copyrighted segments in completions; supports compliance; English scope.
  • Blocklists and custom categories: Enforce organization-specific terms and custom detection; stored encrypted at rest with CMK options.

Minimal working example

Text moderation (Python) Requirements: requests or SDK; endpoint; subscription key or Entra auth.

import requests, json
endpoint = "https://YOUR-CONTENT-SAFETY.cognitiveservices.azure.com"
headers = {"Ocp-Apim-Subscription-Key": "KEY","Content-Type":"application/json"}
payload = {"text": "Sample text", "blocklistNames": []}
r = requests.post(f"{endpoint}/contentsafety/text:analyze?api-version=2024-09-01",
                  headers=headers, data=json.dumps(payload))
print(r.json())
Enter fullscreen mode Exit fullscreen mode

Test: Inspect categoriesAnalysis severities; adjust thresholds; add blocklists.

Checklist

  • Configure separate input/output filters; turn on Prompt Shields for inputs; add groundedness and protected material for outputs.
  • Confirm API rate limits and payload size limits per modality/tier.
  • Use blocklists for brand/PII terms; understand storage and deletion controls.

Where this doesn't apply

Offline moderation pipelines with proprietary classifiers; integrate but consider Content Safety for complementary coverage.

Sources


Top comments (0)