How do Azure AI Content Safety and Prompt Shields defend against jailbreaks and indirect prompt injection?

Azure AI Content Safety and Prompt Shields defend against jailbreaks and indirect prompt injection by applying prompt and content filtering before model execution and downstream actions. These tools detect various user prompt attacks and document attacks in third-party content.

Azure AI Content Safety and Prompt Shields implement several controls to mitigate these risks:

• Prompt and Content Filtering: They detect user prompt attacks such as role-play, encoded bypasses, attempts to change system rules, and fake conversation scaffolding. They also identify document attacks in third-party content like emails, uploaded files, and retrieved grounding data, which aim to manipulate content, gain unauthorized access, exfiltrate data, block capabilities, commit fraud, or spread malware. This aligns with OWASP LLM Top 10 risk LLM01: Prompt Injection.
• Data Handling and Minimization: Controls include minimizing grounding data that reaches the model, preserving access-control context, using permission-aware retrieval, sensitivity labels, DLP, source citation, and response filtering. This helps prevent indirect prompt injection by limiting the attack surface from untrusted data sources.
• Operational Controls: These include rate limits, token and cost ceilings, abuse detection, and integration with security tools like Defender or Sentinel. Throttling model endpoints and agent APIs by tenant, user, app, connector, and action type helps prevent abuse and repeated jailbreak attempts. Security teams are alerted on suspicious prompt patterns, unexpected connector usage, data egress anomalies, and sudden increases in denied tool calls.
• Severity Triage: Microsoft's AI bug bar treats prompt injection enabling zero-click data exfiltration or privileged actions as critical, and similar attacks requiring user interaction as important. This helps prioritize responses to detected threats.