What does the EU AI Act require for autonomous agentic AI systems?

The EU AI Act requires demonstrable controls for autonomous agentic AI systems, which largely overlap with security controls for safety. While the provided sources do not detail specific EU AI Act requirements, they outline a secure-by-construction framework for agentic AI systems that would likely satisfy such requirements.

Key controls for autonomous agentic AI systems include:

• Autonomy boundaries The first architectural decision for an agentic AI system is defining its autonomy level, which dictates subsequent requirements for identity, context, runtime, and data. Autonomy should be multi-dimensional, with each capability having its own setting ranging from fully autonomous to never-permitted, considering factors like reversibility, blast radius, cost, external visibility, and compliance sensitivity. The system should be designed so that an agent cannot perform an action outside its defined autonomy bounds. This aligns with the NIST AI RMF function of Govern by establishing clear policies and procedures for AI system operation.
• Human Oversight & Override Automated defenses are not sufficient; human intervention points must be designed into the architecture for high-stakes, irreversible actions, ambiguous cases, regulatory requirements, or when the agent's confidence is low. This includes pre-action approval gates for high-stakes actions and post-action review queues for sampling completed actions. This directly addresses the OWASP LLM Top 10 risk of LLM07: Excessive Agency by ensuring human control over agent actions.
• Identity & Intent Once autonomy boundaries are defined, it's crucial to establish who is acting and on whose behalf. This involves cryptographic workload identity, delegated authority, and intent attestation to prove what code is running, on whose behalf the agent acts, and what the agent is trying to do. This aligns with the NIST AI RMF function of Protect by ensuring proper authentication and authorization.
• Evaluation and Integration The system should be designed to emit compliance evidence as a byproduct of normal operation. This includes integrating with enterprise SIEM, secrets management, and compliance reporting, and conducting formal threat modeling. This aligns with the NIST AI RMF function of Map by ensuring the system's compliance with relevant regulations.
• Prevention of Autonomy Creep and Ambiguity Implement periodic re-attestation of autonomy levels with explicit sign-off and automated tracking of boundary changes to prevent gradual expansion of autonomy without re-attestation. Use explicit, machine-readable autonomy policies with clear precedence rules to avoid unclear boundaries. This helps mitigate the MAESTRO L3 and L6 threats of Autonomy creep and Autonomy ambiguity.