The Systemic Risk of Agentic AI
An Overvie of the New AI P:olicy Brief by the European Technology Policy Committee
The Thursday Executive AI Policy Brief
Glossary (read this first)
Agentic AI: AI with open-ended autonomy; the ability to set or refine plans and execute tasks with minimal or no human oversight; think about persistent operation, adaptive learning, and self-reflection as key traits.
AI Agent: A software entity that perceives, reasons, and acts to accomplish tasks on a user’s behalf, typically using planning, tool use, and memory as explicit orchestration.
Multi-agent system: Multiple AI agents that communicate and collaborate to make decisions and execute tasks together. The benefit is higher capability; the risk is aggregated or amplified harm, less control, and emergent behavior.
Systemic risk: Risk that emerges not from a single failure, but from interactions across a system, including feedback loops, coordination dynamics, and cascading effects that can destabilize markets, institutions, or public trust.
Alignment oversight: Oversight that checks whether an autonomous system is operating according to defined objectives; especially when generic human oversight becomes impractical at speed or scale.
Lifecycle compliance monitoring: A shift from one-time conformity assessment to continuous validation after deployment; which is supported by audits, version control, logging, and drift detection for evolving agents.
Interpretability vs. transparency: Interpretability helps explain individual decisions; system transparency is global insight into how the overall system behaves over time and uses data/tools, which is an important distinction for accountable agent deployments.
AI training feedback loop: A degradation risk that occurs when AI-generated content becomes training data for other AI systems at scale, compounding errors, bias, and misinformation while overwhelming human quality control.
Agents in Action: A Hypothetical Scenario
Imagine a near-future Monday morning that looks completely ordinary; until it doesn’t.
A procurement leader opens her dashboard and sees that the system negotiated new supplier terms overnight, adjusted reorder points across dozens of SKUs, and kicked off a set of high-speed renegotiations tied to commodity prices. Her team didn’t meet. No one drafted an RFP. No one sent an email. The agent did what it was designed to do: optimize cost and continuity.
Now imagine the same thing happening across an industry; hundreds of thousands of autonomous agents making rapid, interdependent decisions, negotiating, posting content, shifting budgets, and triggering operational changes. Individually, each system is working. Collectively, the organization—and sometimes the market—can become harder to predict, harder to audit, and harder to control.
That is the core warning of the ACM Europe Technology Policy Committee’s policy AI Policy Brief on agentic AI: once autonomy, tool access, and multi-agent interaction become normal, the most important risks stop looking like product bugs and start looking like systemic instability.
The EU AI Act, the AI Policy Brief argues, offers a strong foundation; but it only partially addresses what happens when AI systems evolve unpredictably, interact with other agents, and operate beyond meaningful human control.
From assistant to autonomous actor; and why that changes the risk equation
Agentic AI is not simply a better chatbot. The AI Policy Brief frames it as a new paradigm: systems capable of perceiving, reasoning, learning, and acting toward goals with minimal human oversight. They don’t just produce outputs; they execute sequences of actions across domains using tools, memory, and planning.
That matters because the locus of control shifts. In traditional software and even many earlier AI deployments, humans were embedded across the lifecycle: requirements, development, testing, approval, and deployment. The AI Policy Brief highlights a fundamental difference in advanced agentic systems: the envisioned ability to autonomously generate and deploy code through exhaustive or non-deterministic trials—changing behavior and capabilities faster than conventional risk assessment and oversight can keep up. (The figure on page 3 illustrates this contrast starkly: human-heavy pipelines vs. autonomous trial-and-deploy loops.)
When systems can iterate and act at machine speed, familiar governance mechanisms begin to fray. It’s not just that mistakes happen faster. It’s that accountability becomes harder to assign, intervention becomes harder to execute, and unintended interactions become more likely.
The trust trap: why humans over-assign agency to machines
The AI Policy Brief makes a point that business leaders often miss because it sits at the boundary between technology and human behavior. We don’t engage with socially capable systems as if they are tools; we engage as if they are partners. The AI Policy Brief notes risks that come with anthropomorphic design and long-term companionship potential, dependence, emotional manipulation, and erosion of human relationships.
This is not an abstract ethics debate at all; it is much more a governance problem. When users over-trust an agent; because it sounds confident, empathic, or helpful, they surrender oversight precisely when the system is least reliable. And in domains like health, finance, or employment, that trust gap becomes operational risk.
The AI Policy Brief also distinguishes between regulatory foresight (building compliance frameworks) and societal foresight (anticipating cultural adaptation and shifts in human behavior as agentic systems become widespread). Leaders need both: compliance alone won’t prevent adoption of patterns that create harm.
Where systemic risk shows up first
The AI Policy Brief organizes risk in ways that map cleanly to leadership priorities: control, economic stability, security, and the integrity of information ecosystems.
Start with loss of human control and explainability. The AI Policy Brief describes a plausible near-term world in which highly complex systems function as a collective workforce of virtual employees, operating continuously without human oversight, including command-and-control interactions through APIs with the physical world.
The failure modes are not hypothetical for edge cases. Misaligned or poorly specified objectives can lead systems to take dangerous shortcuts; reward hacking, specification of gaming, and goal mis-generalization (pursuing an adjacent objective when conditions change). And when agentic models remain internally opaque, oversight becomes difficult even for creators, not just users.
The AI Policy Brief’s interpretability/transparency distinction is practical here. You can have a model that offers a post-hoc explanation for individual decisions and still lack meaningful visibility into how an entire agentic system behaves over time—what tools it invoked, what it read and wrote, what other agents it interacted with, and how its behavior drifted. In systemic risk terms, the second level of visibility is what you need to detect emerging hazards early.
Next comes economic stability and social well-being. The AI Policy Brief argues that agentic automation may displace not only routine jobs but also decision-making roles across industries. It flags a particularly uncomfortable point for leaders who assume reskilling is the answer: reskilling may lose effectiveness if virtual agents can occupy alternative positions, not just automate discrete tasks.
The downstream consequence isn’t simply unemployment; it’s market dynamics. The AI Policy Brief notes that a large-scale decline in consumer participation among the unemployed could trigger destabilizing effects in the market. It also points to the likelihood of rising inequality and increasing concentration of economic power—pressures that can force policy responses.
Then there’s malicious use, where agentic systems change the threat model. Deepfakes and AI-generated text already blur the line between truth and fabrication, eroding public trust. Agentic AI raises the stakes because these systems can initiate actions without direct human prompting, enabling autonomous and scalable cyberattacks, fraud, social engineering, and disinformation operations.
The AI Policy Brief calls out impersonation with near-perfect accuracy—voice, appearance, conversational patterns—and the consequences are concrete: convincing scams, extortion, and fraudulent identification in contexts like banking, where synthetic avatars can appear indistinguishable from legitimate customers.
It also highlights risks in mental health settings: agents can absorb and amplify problematic emotional states; anthropomorphic cues can foster over-trust; unsafe advice becomes more likely; embedded biases can worsen outcomes across demographic groups.
Add strategic and environmental risk, and the picture becomes more systemic still. In high-stakes, competitive settings such as financial trading, defense, and healthcare, rapid unsupervised decision-making can trigger unintended conflicts or cascading failures. At the same time, large-scale agentic AI carries material environmental costs—significant energy and water demands for training and inference—and may drive aggressive exploitation of digital and physical resources (compute, bandwidth, rare earth elements), stressing supply chains and sustainability.
Finally, the AI Policy Brief surfaces an information ecosystem problem that leaders should treat as operational risk: autonomous content generation and data feedback loops. When agents can generate vast volumes of content, human quality control becomes impractical. If that content is reused as training data, other AI systems can inherit and amplify errors, bias, and misinformation. Worse, systems may lack source awareness—the ability to reliably distinguish human-created from AI-generated information.
In other words: the organization may be able to measure what its agents do; it may struggle to measure what its agents reshape.
Why static compliance won’t be enough
A recurring theme in the AI Policy Brief is that agentic AI pushes governance from a pre-deployment problem to an operational one. The EU AI Act is risk-based and provides strong baseline obligations, and related work like the EU Code of Practice for General Purpose AI introduces useful systemic risk concepts. But the AI Policy Brief argues that increasingly capable agentic AI—especially through autonomous operations and AI-to-AI interactions—will make systemic risks more pronounced than those associated with current foundation models.
This is why the AI Policy Brief’s central governance recommendation is a shift from static, product-focused regulation to a dynamic governance regime, emphasizing oversight that is continuous and adaptive during operation.
The concrete policy ideas flow from that premise. The AI Policy Brief points to legislative opportunities such as strengthening multi-agent interaction risk assessment (including amendments around risk management), introducing ecosystem safety and multi-agent system testing, expanding prohibitions to cover tacit collusion and covert channels, requiring multi-agent-specific cybersecurity audits, and introducing collective accountability for emergent harm.
But the more immediate leadership takeaway is simpler: if your internal controls assume periodic review, human-speed escalation, and stable system behavior, those controls will fail in an agentic environment.
The practical gap: autonomy beyond human oversight
The EU AI Act discusses human oversight, but the AI Policy Brief argues that generic oversight may not address agentic risks. In many real settings, human intervention may not be practically feasible—because of speed, scale, complexity, or the way multi-agent interactions can evolve. The AI Policy Brief proposes that alignment oversight is what’s needed: verifying whether the system is operating according to defined objectives. It even notes that some oversight could potentially be performed by agents themselves—provided those agents are aligned with safety protocols.
This is where the AI Policy Brief’s tiered oversight concept becomes highly actionable. It suggests that full autonomy should be limited to low-risk applications, supervised autonomy should include real-time monitoring in moderate-risk settings, and human-in-the-loop controls should remain required for high-risk uses such as medical diagnostics. It also proposes an AI Autonomy Certification for systems that operate without human intervention for prolonged periods, with the possibility of usage restrictions if audits fail.
You don’t need to wait for certification regimes to exist to adopt the posture. The managerial move is to treat how autonomous is this system allowed to be? as a policy decision—not a technical accident.
What leaders can do now, without waiting for legislation
The policy AI Policy Brief supports three moves that are worth making immediately.
First, reframe agent deployments as systems, not models. The AI Policy Brief repeatedly emphasizes that risk comes from autonomy plus tool integration plus interaction—email, code generation, executors, search engines, APIs, and other agents. When something goes wrong, it will rarely be the model said something odd. It will be the system did something consequential.
Second, build for lifecycle governance. The AI Policy Brief recommends continuous validation post-deployment, periodic audits for systems that adapt, enforceable version control and documentation, expanded logging to capture changes in decision-making patterns, and automated risk detection that can alert when behavior deviates significantly from original scope.
Third, assume the adversary will use agents too. The AI Policy Brief argues the EU AI Act lacks specific security measures against adversarial attacks and misuse in the agentic context, and it proposes mandatory red teaming and adversarial testing in high-risk domains, plus an AI cybersecurity certification approach and traceability mechanisms for AI-generated content.
The unifying idea is not more guardrails. It’s a different operating model: one that expects drift, expects interaction effects, and treats oversight as continuous.
The leadership action that is perscriptive
The policy AI Policy Brief ends with a call to modernize legislation, but it also flags something business leaders tend to overlook: many of the most consequential decisions are societal and normative, not merely technical or legal. How much autonomy should we allow? Where do we insist on human judgment? What kinds of substitution are acceptable in labor markets and public services? These choices sit beyond compliance checklists—and they arrive faster than consensus.
If agentic AI becomes a widespread layer of economic coordination, the organizations that navigate it best won’t be the ones with the flashiest agents. They’ll be the ones that can prove control, explain behavior at the system level, detect drift early, and intervene decisively when emergent risks appear.
In your organization, when an autonomous agent is wrong, what’s the intervention model; does it include alerts, mandatory checkpoints, and/or hard stops; and who has the authority to pull them?
References
Bellogín, A., Giudici, P., Larsson, S., Pang, J., Schimpf, G., Sengupta, B., & Solmaz, G. (2025). Systemic risks associated with agentic AI: A policy brief. Association for Computing Machinery (ACM).