How do I run a continuous red-teaming program for AI agents after launch?

A continuous red-teaming program for AI agents after launch requires integrating security evaluation into the CI/CD pipeline, utilizing automated tools, and performing production evaluations. This approach ensures ongoing identification and mitigation of vulnerabilities, addressing the NIST AI RMF function of Govern and Map.

• Integrate into CI/CD Pipeline: Security evaluation must be continuous, automated, and integrated into the same CI/CD pipeline that ships application code. This prevents regressions from being merged and ensures that every change to prompts, models, tools, or policies is evaluated against a golden dataset. This addresses the OWASP LLM Top 10 risk of LLM01: Prompt Injection and LLM02: Insecure Output Handling by continuously testing for these vulnerabilities.
• Maintain a Golden Dataset: Establish a curated golden dataset of inputs covering the security and safety surface with expected behaviors. This dataset should include known prompt injection variants, jailbreak attempts, edge cases from past bugs, representative legitimate inputs, and inputs that probe specific policy boundaries. The dataset should grow over time with new findings.
• Utilize Robust Evaluation Harnesses: Employ an evaluation harness to measure key metrics such as task success rates, refusal rates, tool selection quality, cost per task, latency, and consistency. This multi-dimensional evaluation prevents shipping changes that might improve one metric but negatively impact overall performance or security.
• Automated Adversarial Input Generation: Run automated red-teaming tools, such as Garak or PyRIT, on every release candidate to generate adversarial inputs at scale. The output from these tools should feed back into and expand the golden dataset. This directly addresses the NIST AI RMF function of Measure by continuously assessing the AI system's resilience to adversarial attacks.
• Production Evaluation: Close the loop with production evaluation methods like shadow-mode evaluation, where new versions run against production traffic without affecting users, and canary deployments, which expose a small fraction of traffic to watch for anomalies. Online metrics from these deployments should feed back into the evaluation suite, blurring the line between testing and production. This aligns with ISO/IEC 42001 control A.7.2.1 AI system monitoring by continuously monitoring the AI system's performance and security in a live environment.