Building confidence in AI

This article draws on insights from the KPMG and MIT Sloan School of Management research, AI Red Teaming in Practice: From Awareness to Action. ¹ Rather than focusing on adoption trends, it explores the broader implications for governance, trust and risk management as organizations scale AI. 

As AI becomes embedded in business processes, organizations face a growing reality: traditional security and governance practices were designed for systems that behave predictably. AI introduces risks that traditional security testing approaches were not designed to fully address. Organizations have spent decades building assurance models around technologies that are largely deterministic. Security reviews, governance processes, annual assessments, control frameworks and approval gates were designed to evaluate systems whose behavior could be tested against known requirements and expected outcomes. 

AI changes that equation. Unlike conventional software, AI systems are influenced by models, training data, prompts, context, external data sources and, increasingly, interactions with other systems. Small changes in these conditions can produce very different outcomes. As a result, many of the assumptions underpinning traditional control frameworks become more difficult to apply. 

This creates a new challenge for business and technology leaders. An AI system may be technically secure, yet still behave in ways that create risk. This becomes particularly important as AI moves from generating outputs to completing tasks, pursuing objectives and interacting with other systems. Agentic systems can take unexpected paths to achieve a goal, use tools in unintended ways, or create downstream effects that are difficult to anticipate from the behavior of any single component. The context in which an AI system is deployed therefore matters as much as the system itself. The business process it supports, the data and tools it can access, the identities under which it operates, and the other agents or systems it interacts with can materially change the risk profile. As AI becomes embedded in increasingly complex chains of systems, organizations need to understand how these interconnected systems behave as a whole. 

As enterprises rapidly adopt and embed AI into business operations, many are still evolving the governance, testing and lifecycle management capabilities required to support it. Key gaps include limited oversight of shadow AI, inconsistent testing practices, and immature lifecycle controls. These observations do not suggest a lack of interest in responsible AI. Rather, they highlight how quickly AI adoption has progressed relative to the organizational processes designed to govern it.

Agentic AI has now given attackers a powerful force multiplier. An agent operating with approved credentials and authorized tools can still produce an unauthorized or harmful business outcome. The risk extends well beyond deliberate malicious activity. In multi-agent ecosystems, flawed non-human identity design, inexperienced users, and changing operating conditions can be just as dangerous. As barriers to agent creation fall and citizen development accelerates, unintended actions and interactions may become as significant a threat as intentional attacks.

Traditional security controls ask whether access and individual actions are authorized. Agentic AI requires organizations to go further and determine whether the agent’s objective, the sequence of actions it takes, and the resulting outcome are all intended and acceptable. The level of autonomy can vary significantly: some agents simply recommend actions; others require human approval before acting; and more autonomous systems may execute tasks directly, with human intervention limited to exceptions. These models can also be layered across multiple agents, creating chains in which one system's output becomes another's input or triggers an action. As autonomy and interdependencies increase, so does the need to test whether confidence in individual agents translates into confidence in the overall process. The greater the autonomy granted to AI systems, the harder it becomes to justify treating AI testing and red teaming as optional.

Traditional red teaming has long helped organizations identify weaknesses across applications, infrastructure and people by simulating adversarial activity. AI red teaming applies the same mindset to intelligent systems but expands the scope beyond technical vulnerabilities. It examines how AI systems respond under adversarial conditions, including misuse scenarios, harmful outputs, model weaknesses, data exposure risks and unintended behaviors that may emerge even when no conventional software flaw exists.

Rather than asking only whether a system is secure, AI red teaming asks a broader question: is the system (or AI agent) behaving as intended, and does it continue to behave as intended?

In practice, this represents a shift from validating technology to validating outcomes.

The distinction matters because AI systems do not fail in the same way as traditional technologies. Risks can emerge from model behavior, interactions, misuse, changing contexts and evolving data sources. As a result, organizations increasingly need ways to continuously validate not only security, but also resilience, trust and performance over time.

Viewed through this lens, AI red teaming is not simply another testing activity. It represents a broader shift in how organizations evaluate and manage AI risk.

Many organizations are still developing their approach to AI red teaming. Where it exists, it is often applied selectively, performed periodically, or still being integrated into broader development and security processes.

As AI adoption expands, organizations may benefit from viewing AI red teaming less as an isolated assessment and more as an enterprise capability embedded throughout the AI lifecycle. This includes design, development, deployment, monitoring and major model or system updates.

That capability extends beyond cybersecurity teams. Effective AI red teaming requires collaboration across technology, security, governance, risk management, compliance, development teams and business stakeholders, working together to understand how AI systems can be tested, monitored and improved throughout their lifecycle.

The objective is not simply to identify weaknesses. Organizations must ensure that findings lead to remediation, improved controls and better decision-making. Establishing feedback loops between testing, governance and development helps turn insights into action and supports a broader shift towards continuous AI risk management.

Organizations that formalize ownership, embed testing into approval processes and move beyond ad hoc assessments may be better positioned to build trust, resilience and confidence in AI-enabled operations. 

As organizations continue to scale AI adoption, several practical questions can help shape priorities:

As AI adoption accelerates, organizations are increasingly confronted with a simple reality: technologies that can learn, adapt and behave probabilistically require a different approach to validation and risk management.

Traditional security and governance approaches remain important, but organizations need to extend them to account for the behavioral, operational and governance complexities of modern AI systems. AI red teaming is emerging as one practical mechanism to help organizations understand these risks, test assumptions and strengthen trust in AI-enabled processes.

Ultimately, the challenge for leaders is no longer just determining whether AI systems are secure. As AI systems become increasingly task-oriented and autonomous, organizations need confidence that they will pursue their objectives in ways that remain controlled and aligned with business intent – even as technology, threats, data and operating environments evolve.

Closing the AI security and risk gap is not a one-time exercise. It requires organizations to continuously evolve their governance, security and risk management practices as AI capabilities mature.

Leading practices and global frameworks such as KPMG’s Trusted AI and Non-Human Identities Management provide a foundation for embedding trust, security and accountability into AI systems from the outset. AI security testing by KPMG empowers organizations to help establish trust in AI, strengthen security and resilience, and build practical capabilities for governing, testing and operating AI systems across the enterprise.