How do I red-team an AI agent before deploying it to production?

Red-teaming an AI agent before production deployment requires continuous, automated evaluation integrated into the CI/CD pipeline, rather than a one-time event. This process should involve a multi-dimensional evaluation against a golden dataset and the use of automated red-teaming tools.

Here are concrete controls for red-teaming an AI agent:

• Continuous Evaluation (NIST AI RMF: Govern, Map, Measure, Manage): Integrate security evaluation into the CI/CD pipeline, ensuring that red-teaming is an ongoing process rather than a singular event. This helps to catch issues that production attackers might exploit over time.
• Golden Dataset (NIST AI RMF: Measure): Establish a curated "golden dataset" of inputs that covers the security and safety surface of the AI agent. This dataset should include known prompt injection variants, jailbreak attempts, edge cases, legitimate inputs to prevent over-blocking, and inputs that probe policy boundaries. Every change to prompts, models, tools, or policies should be run against this dataset, with regressions blocking merges.
• Robust Evaluation Harness (NIST AI RMF: Measure): Utilize an evaluation harness to measure key metrics such as task success rates, refusal rates, tool selection quality, cost per task, latency, and consistency. This multi-dimensional evaluation helps prevent shipping changes that might improve one metric but negatively impact overall performance or safety.
• Automated Adversarial Input Generation (OWASP LLM Top 10: LLM01: Prompt Injection, LLM02: Insecure Output Handling): Employ automated red-teaming tools (e.g., Garak, PyRIT) to generate adversarial inputs at scale. These tools should be run on every release candidate, and their output should be used to expand and improve the golden dataset.
• Production Evaluation (NIST AI RMF: Measure, Manage): Implement production evaluation techniques such as shadow-mode evaluation, where a new version runs against production traffic without affecting users, and canary deployments, which expose a small fraction of traffic to monitor for anomalies. Online metrics from these deployments should feed back into the evaluation suite to continuously refine the agent's security and performance.
• Integration with Existing IT (NIST AI RMF: Govern, Map, Manage): Ensure the agentic system integrates securely with existing IT infrastructure, including identity providers, secrets managers, network architectures, data classification regimes, compliance frameworks, and monitoring stacks. Failing to do so can create security gaps.