What has the Microsoft AI Red Team learned about attacking and defending AI systems?

The Microsoft AI Red Team has learned that securing AI systems is a continuous, system-specific, and harm-driven process that requires a defense-in-depth approach, moving beyond shallow benchmarking to understand system capabilities and deployment contexts. They emphasize that LLMs amplify existing security risks and introduce new ones, and that even simple prompting can break systems without gradient attacks.

Here are concrete controls and practices based on their findings:

• Implement identity and data governance by inheriting existing controls like Microsoft Entra ID, Conditional Access, role-based access control, sensitivity labels, Microsoft Purview data loss prevention, audit logging, and tenant isolation. This aligns with the NIST AI RMF function of Govern and the OWASP LLM Top 10 risk of LLM01: Prompt Injection by ensuring agents do not bypass established permissions.
• Address prompt injection and poisoned content by using Prompt Shields (Azure AI Content Safety) for user prompt attacks and indirect attacks, and by employing data marking (Spotlighting) to differentiate external data from instructions. For agent systems, use explicit provenance labels, delimiter schemes, structured tool schemas, and policy checks to prevent retrieved data from creating new instructions or authorizing tool use. This directly addresses LLM01: Prompt Injection and LLM02: Insecure Output Handling.
• Apply robust controls for agents executing code or calling APIs, including sandboxing, egress controls, output validation, and human approval for irreversible side effects. Maintain tamper-resistant traces of user actions, prompts, retrieved sources, agent versions, tool arguments, approval decisions, and results. This relates to the NIST AI RMF function of Protect and the OWASP LLM Top 10 risk of LLM04: Insecure Plugin Design.
• Conduct continuous, harm-driven red teaming that is specific to the system's capabilities, data access, users, adversaries, and deployment context. Use automation tools like PyRIT to expand coverage, but preserve expert human judgment for assessing business impact and subtle policy violations. Evaluate both direct and multiturn attacks, as individually benign turns can accumulate into harmful trajectories. This aligns with the NIST AI RMF function of Map and Measure.
• Prioritize vulnerabilities based on MSRC-style severity, focusing on issues that allow data exfiltration, privileged action, cross-tenant exposure, code execution, credential theft, or durable memory poisoning. Retest after model updates, prompt changes, connector additions, permission changes, and data-source migrations. This aligns with the NIST AI RMF function of Measure and Manage.
• Implement least privilege for AI agents at three levels: the user, the agent, and the tool. This means the user must be authorized for the data and action, the agent limited to its approved scenario, and the connector/API token exposing only necessary operations. Use separate identities for agents and require step-up authorization for sensitive actions. This directly addresses AML.M0026 Privileged AI Agent Permissions Configuration and AML.M0027 Single-User AI Agent Permissions Configuration.