Beyond governance: How Internal Audit builds real trust in AI

Artificial Intelligence (AI) has moved beyond experimentation. It is increasingly embedded in core business processes, supporting decisions, automating workflows, generating content, detecting anomalies, predicting outcomes and, in some cases, acting with growing autonomy. Linked to this, regulatory developments and public scrutiny are converging toward the same conclusion: high-impact AI must be demonstrably trustworthy. Organizations that cannot explain how AI decisions are produced, or how risks are controlled in practice, will struggle to maintain stakeholder trust.

This shift has profound implications for Internal Audit. Traditional audit approaches were often built around relatively stable IT systems, defined business processes, and clear control points. AI challenges that logic. AI systems can rely on large and dynamic datasets, evolve over time, produce probabilistic outputs, and interact with business users in ways that are difficult to predict fully upfront. Generative AI and agentic AI further increase this complexity by enabling systems to interpret prompts, generate content, recommend actions, or even orchestrate tasks across systems.

AI is also broader than many organizations realize. It may include custom-built models, vendor-specific tools, AI

features embedded in enterprise platforms, online AI tools, and unregistered “shadow AI,” all of which can be used in different ways (decision AI, generative AI and, increasingly, agentic AI). Risks differ depending on how AI is accessed and used.

For Internal Audit, this creates both urgency and opportunity. Internal Audit can help organizations move from high- level confidence, based solely on policies or governance frameworks, to evidence-based assurance: demonstrating that AI systems behave as intended, that risks are understood and mitigated, and that controls remain effective in practice. While AI also offers powerful opportunities to enhance the audit function itself, this article focuses on the other side of the equation: how Internal Audit can provide assurance over the AI used across the organization.

The question for organizations is no longer simply “Are we using AI?” but rather “Can we demonstrate that our AI is governed, controlled, and trustworthy in practice?"

Olivier Elst

Partner | Advisory

KPMG in Belgium

mail
call

AI risks are affecting all organizations

AI is transforming industries at an unprecedented pace, bringing both game-changing opportunities and profound risks that no organization can afford to ignore. As AI continues to evolve, organizations must navigate a complex landscape of risks that have the power to disrupt operations and erode trust, including:

Strategic risks arise when AI negatively impacts the organization's long-term objectives, reputation, or competitive position. Incorrect AI- driven decisions, biased outputs, loss of stakeholder trust, or an inability to scale AI responsibly can undermine business performance and damage brand value. Organizations must ensure that AI creates sustainable value while protecting their reputation and market position.

Compliance risks arise when AI systems fail to meet legal, regulatory, or contractual requirements. With regulations such as the EU AI Act now taking effect, organizations must ensure that AI systems are transparent, explainable, properly documented, and compliant with applicable laws relating to privacy, intellectual property, consumer protection, and sector- specific requirements. Failure to do so can result in fines, legal disputes, and regulatory scrutiny.

Governance and ethical risks include unclear ownership, biased or unfair outcomes, inadequate human oversight, weak governance structures, and a lack of AI literacy among users and decision-makers. Effective governance is essential to ensure that AI is used responsibly, transparently, and in alignment with organizational values and risk appetite.

Operational risks relate to the reliability, security, and effectiveness of AI systems in day-to-day operations. These risks include cyberattacks, data breaches, poor data quality, inaccurate or unstable AI outputs, and inadequate oversight of third-party AI providers. If not properly managed, operational failures can disrupt processes, impair decision-making, and create significant financial and business consequences.

Across industries, organizations increasingly want confidence that their AI systems are:

Secure - protected against misuse and adversarial attacks.
Compliant - aligned with evolving regulatory expectations.
Transparent and accountable - with clear traceability and ownership.
Reliable and explainable - appropriate for their decision context.
Scalable - in a responsible way, without losing control as adoption grows.

Meeting these expectations requires more than policy reviews. It requires structured, evidence-based insight into how AI systems operate in reality.

The role of Internal Audit: moving beyond governance-only assurance

Traditionally, many organizations have started their AI journey by establishing governance frameworks, policies, oversight committees, and approval processes. While these foundations are essential, a common misconception is that strong governance automatically translates into trustworthy AI. In reality, governance only provides the framework within which AI operates. The true risks often emerge within the AI systems themselves: from data quality issues and biased outcomes to inaccurate predictions, security vulnerabilities, and a lack of transparency around how decisions are made.

This is where Internal Audit can provide significant value. Rather than limiting assurance activities to governance structures and policies, Internal Audit can help organizations assess whether AI systems are operating as intended, whether risks are adequately mitigated, and whether controls remain effective throughout the AI lifecycle. To achieve this, Internal Audit should adopt an end-to-end perspective that considers the complete AI lifecycle: from design and development to deployment, monitoring, and ongoing use. This includes evaluating not only governance, policies, and oversight committees, but also the quality of data used by AI systems, the robustness of model development practices, the effectiveness of human oversight mechanisms, and the organization’s ability to detect and respond to emerging risks.

Equally important is recognizing that AI is not a single technology. Organizations are increasingly relying on a diverse landscape of AI solutions, including traditional machine learning models, generative AI solutions, AI embedded within third-party software, and emerging agentic AI systems. Each of these technologies introduces distinct risk profiles and therefore requires a tailored assurance approach.

In practice, Internal Audit can add value in two ways. For organizations beginning their AI program, Internal Audit may perform “in-flight” or “pre-assurance” assessments; for more mature organizations, Internal Audit can evaluate the effectiveness of implemented controls, including security and privacy reviews, AI governance model reviews, and post-deployment reviews.

Ultimately, the objective of AI auditing is not to slow down innovation, but to support it. By moving beyond governance-only reviews and providing independent assurance over the design, implementation, operation, and monitoring of AI systems, Internal Audit can help organizations unlock the benefits of AI while ensuring that risks remain within acceptable boundaries.

Three types of AI audit Internal Audit should consider

A mature AI audit plan should not rely on one single audit type. Typically, it should start by addressing AI Governance, often in a first instance testing the comprehensiveness of AI governance across all types of AI, and with a 360° view on all governance aspects. As part of the AI governance audit, or as a separate audit an EU AI Act Compliance Audit can be held in order to assess whether the organization is ready to meet applicable regulatory obligations. Finally, detailed assurance on a selection of specific AI solutions (at first addressing a mix of various types of AI systems) could be considered.

1. AI governance audit: Is the organization structurally ready to manage AI?

An AI governance audit evaluates whether the organization has the structural components required to govern AI responsibly and effectively. This includes vision and strategy, roles and responsibilities, governance bodies, policies and procedures, risk management, AI inventory, data management, security, privacy, documentation, monitoring, and AI literacy.

KPMG’s AI governance framework gives a comprehensive overview across all AI domains covering governance, risk management, and compliance. Structured around nine core dimensions, it ensures consistent implementation of policies, controls, and oversight.

This type of audit is particularly relevant when organizations are scaling AI but still rely on fragmented governance. Common findings may include unclear ownership, incomplete AI inventories, weak approval processes, insufficient documentation, limited training, or poor visibility over third-party and embedded AI. The value of the audit is that it gives management and the audit committee a clear view of whether the organization has a scalable foundation for AI risk management.

A practical AI governance audit typically starts by defining the scope and relevant governance dimensions, reviewing policies and documentation, mapping the AI landscape, conducting interviews or workshops, assessing maturity per dimension, identifying gaps, and translating findings into a prioritized roadmap. Where relevant, the audit can also specifically zoom in on third-party risk management, evaluating how AI risks introduced by external vendors and embedded solutions are governed and controlled.

2. EU AI Act compliance audit: Is the organization ready to demonstrate compliance?

An EU AI Act compliance audit assesses whether the organization is ready to meet applicable regulatory obligations. This is especially relevant where the organization develops, deploys, or relies on AI systems that may fall under AI Act risk categories.

This type of audit should begin with visibility: the organization needs an AI inventory. Without a clear view of AI systems, models, providers, users, purposes, and data flows, it is difficult to classify risk, assign obligations, or evidence compliance. The next step is risk classification, followed by an assessment of documentation, governance processes, control design, human oversight, monitoring, incident handling, and AI literacy requirements.

Importantly, an EU AI Act compliance audit should be integrated into broader AI governance and assurance. If treated as a standalone legal checklist, it may miss operational risks. If embedded into a Trusted AI control framework, it becomes more valuable: it helps the organization connect regulatory obligations with practical governance, control, and monitoring mechanisms.

3. AI solution audit: Can the AI system itself be trusted?

An AI solution audit focuses on an individual AI system or group of systems. It asks a more targeted question: Is this AI solution designed, implemented, used, and monitored in a controlled way?

We developed a structured AI assurance framework, along with our trusted AI framework, which integrates testing of governance aspects and internal controls with quantitative model testing to ensure a thorough AI audit.

On the qualitative side, we apply a risk-based approach which focuses on organizational governance, model design, risk exposure, and alignment with responsible AI principles on a process level.

Equally important is our quantitative AI testing, which introduces hands-on, data-driven analysis using statistical and performance-based methods. These include, for example:

Fairness checks to detect and address bias.
Class imbalance analysis to ensure equitable model performance.
Robustness testing to evaluate model stability under varied conditions.

By combining governance, technical, and operational perspectives, AI solution audits provide organizations with assurance that their AI systems are not only compliant and well-controlled but also deliver reliable and trustworthy outcomes in practice.

Large Language Model (LLM) solutions audit

Generative AI solutions, such as chatbots, copilots, and document- generation tools, introduce risks that are distinct from traditional technology solutions. Unlike deterministic systems, Large Language Models (LLMs) generate probabilistic outputs that may vary depending on context, prompts, and user interactions. This creates risks related to hallucinations, inappropriate content generation, prompt manipulation, information leakage, and overreliance on AI- generated outputs.

An audit of an LLM solution therefore goes beyond traditional IT controls and evaluates four key areas: (i) governance and lifecycle management, (ii) security, privacy and data protection, (iii) technical performance and reliability, and (iv) business use and human oversight.

AI agent audit

Agentic AI differs by being dynamic and adaptive, capable of making real-time decisions, continuously learning from its environment, and autonomously adjusting to changing conditions. As AI agentic systems proliferate and scale, a structured framework is essential to understand and categorize them based on their capabilities. Using our TACO framework, we can assess them by examining the complexity of the goals they fulfil, the depth of planning required, and the level of coordination and orchestration involved. Below is an overview of agent categories and scope.

Machine Learning (ML) solutions audit

Traditional ML models present a different risk profile. Rather than generating content, these models are often used to support predictions, classifications, forecasting, and automated decision-making. Risks primarily relate to data quality, model design, assumptions, performance, bias, and ongoing monitoring. When auditing these solutions, the audit may focus on model design, assumptions and limitations, data quality, sensitivity testing, performance monitoring, model output, and use.

Six elements for successful audit functions

For Internal Audit functions, the challenge is to create an AI audit approach that is sufficiently robust without becoming overly theoretical. A pragmatic starting point is to build a phased AI audit roadmap:

Create visibility over AI use

Start with an AI landscape and inventory, including custom-built AI, vendor AI, embedded AI, online tools, and shadow AI.
Assess governance foundations

Review roles, policies, approval processes, risk oversight, documentation, and AI literacy.
Prioritize high-risk use cases

Focus on AI systems with material impact on customers, employees, financial reporting, compliance, operations, or public trust.
Perform solution-level audits

Test whether selected AI systems are controlled across design, data, technical implementation, use, monitoring, and output validation.
Connect regulatory readiness with operational controls

Translate EU AI Act and GDPR expectations into practical controls, evidence, and ownership.
Embed continuous monitoring

AI assurance should not stop at deployment; model performance, drift, incidents, user behavior, and control effectiveness need ongoing attention.

Auditing AI is not about slowing down innovation. It is about enabling responsible innovation. Organizations want to benefit from AI, but they also need confidence that AI systems are secure, compliant, transparent, reliable, and aligned with their intended purpose.

The most valuable AI audits will therefore be those that combine governance insight, regulatory understanding, technical testing, and business context. They will not only ask whether an AI policy exists, but whether AI systems are actually controlled. They will not only check whether use cases are approved, but whether outputs are traceable, monitored, and explainable. They will not only review compliance readiness, but whether the organization can defend its AI decisions when challenged.

For Internal Audit, AI represents a defining opportunity. By moving from governance-only reviews to evidence-based AI assurance, Internal Audit can help organizations build trust in AI - not as a slogan, but as a demonstrable control reality.

How can KPMG help?

By combining governance, technical assurance, cybersecurity expertise and regulatory knowledge, KPMG helps organizations to deploy AI with confidence, ensuring that innovation is supported by trust, transparency, and effective risk management.

Experience in AI auditing: KPMG combines expertise in Internal Audit, AI, risk management, cybersecurity, and regulation to help organizations assess AI risks and establish effective assurance approaches. Our experience across a broad range of AI technologies enables us to identify practical risks and provide actionable recommendations.
Tailored AI solutions: Every organization's AI landscape is different. We apply a risk-based approach and tailor our audit procedures to the specific AI use cases, technologies, risk profile, and regulatory requirements of your organization.
Strengthened AI governance: We help organizations assess and enhance their AI governance framework by evaluating policies, accountability structures, oversight mechanisms, and risk management processes. This helps establish a solid foundation for responsible AI adoption and regulatory compliance.
Enhanced cybersecurity: AI introduces new security threats, ranging from data leakage and unauthorized access to prompt manipulation and model attacks. We assess the effectiveness of existing safeguards and help strengthen the resilience of AI systems and supporting infrastructure.
Skilled audit teams: Many Internal Audit functions face a shortage of AI-specific knowledge and methodologies. Through co-sourcing, training and subject matter support, KPMG helps audit teams build the capabilities needed to independently assess AI risks and controls.