Black-Box Red Teaming of Agentic AI: A Taxonomy-Driven Framework for Automated Risk Discovery
The paper proposes a black-box framework for automatically red-teaming agentic AI systems, using a seven-domain risk taxonomy, generated adversarial multi-step scenarios, and LLM-judge plus human-validated evaluation to uncover failures in agents that use tools and operate autonomously. Its useful angle is treating agent security as an architectural, behavior-level problem rather than a chat-model prompt-safety problem, showing high governance, privacy, and agent-behavior risks across CrewAI and AutoGen setups, though the evidence is still limited by the small set of agent frameworks/base models and reliance on generated scenarios and LLM-based judging.
Abstract
Agentic systems are rapidly moving to production, where they read untrusted inputs, call tools with real permissions, and act autonomously, expanding the security surface beyond chat-only models. Yet standard evaluations remain single-turn and fail to capture multi-step agent vulnerabilities. We present a systematic black-box framework for risk-aware agent evaluation requiring only basic system descriptions. Our approach introduces: (1) a seven-domain taxonomy mapping observable behaviors to risk categories, (2) fully automated SAGE-RT red teaming producing 120 adversarial scenarios per domain, and (3) human-validated evaluation using LLM judges. Empirical validation across two agent architectures (CrewAI and AutoGen) with four base models reveals alarming patterns: 56.25\% average governance risk, 65\% privacy risk in multi-agent configurations, and agent behavior vulnerabilities reaching 85\%. Our black-box approach effectively identifies critical architectural vulnerabilities without privileged access, providing a scalable path toward safer agent deployments.