What key metrics and KPIs should I monitor for a production AI agent?

For a production AI agent, key metrics and KPIs should focus on behavioral telemetry, security, and operational performance, extending beyond traditional application monitoring to capture agent-specific actions and decisions.

• Behavioral Telemetry: Monitor what the agent is attempting to do, its decision-making process, the sequence of actions across workflows, and whether the outcome aligns with the intended purpose. This includes logging decisions and tracing AI behavior, which is a mechanism for transparency and accountability (NIST-MEASURE-2.8). Distributed tracing with a stable trace ID propagated through every hop is essential for forensic replay and reconstructing past tasks.
• Security and Resilience: Evaluate and document the AI system's security and resilience, including adversarial robustness, prompt-injection resistance (OWASP LLM01), and abuse resistance (OWASP LLM04). Monitor for log tampering (MAESTRO L5, L6) by ensuring tamper-evident audit logs and out-of-band shipping to a SIEM. Also, track for PII leakage through logs (MAESTRO L5, L6, L2) and implement configurable redaction at ingestion.
• Operational Performance and Cost: Track task success rates, refusal rates, tool selection quality, cost per task, latency, and consistency. Implement cost anomaly detection to identify runaway agent loops or adversarial exploitation that could lead to substantial bills (MAESTRO L5, L4).
• Continuous Evaluation: Integrate continuous, automated security evaluation into the CI/CD pipeline, running against a golden dataset that includes known prompt injection variants, jailbreak attempts, and edge cases. This helps track identified and emergent risks over time (NIST-MEASURE-3.1).
• Intervention and Control: While not a metric, the ability to intervene at runtime to prevent unsafe actions or policy violations is crucial. This involves mechanisms ranging from soft guidance to hard stops, acting at decision time rather than after the fact.