🚨 Your AI System Was Approved. Nobody Checked If the Controls Worked.

Your organization has an AI governance framework.

Every AI system undergoes a risk assessment. Legal, Risk, and Technology review the documentation. Required controls are identified. Approvals are recorded.

Everything looks good on paper.

Then an AI-powered customer service agent starts giving customers incorrect information. An automated decision-support system produces outcomes that employees cannot adequately explain. A vendor updates its underlying model, introducing new risks that nobody anticipated.

When the governance team investigates, it discovers something uncomfortable.

The system was approved. The required controls were documented. The appropriate stakeholders signed off.

But nobody had verified whether those controls actually worked.

This is one of the most consequential gaps in enterprise AI governance: confusing the existence of governance controls with evidence of their effectiveness.

And it raises an important question.

Are organizations governing AI risk, or simply documenting that governance activities occurred?

As enterprises formalize AI governance, they naturally adopt familiar mechanisms: policies, risk assessments, approval workflows, committees, and documentation requirements.

These are necessary components of an effective governance program.

But they can also create a false sense of security.

Consider two organizations evaluating an AI system that will help employees make decisions about customer eligibility.

Organization A: The Documentation Approach

Organization B: The Control Effectiveness Approach

Both organizations can demonstrate that governance activities occurred.

But Organization B has taken the additional step of generating evidence that its controls can manage the risks they were designed to address.

The distinction matters because a documented control is not necessarily an effective control.

Requiring human oversight does not guarantee meaningful human intervention.

Requiring bias testing does not establish that testing was sufficiently representative.

Requiring monitoring does not mean anyone will detect or respond to deteriorating performance.

And requiring an incident response procedure does not demonstrate that the organization can execute it.

The real question is not whether the control exists.

It is whether the control works under the conditions in which the AI system actually operates.

An AI risk assessment identifies potential risks and assigns mitigation measures.

Human oversight is required. Performance testing is documented. Access restrictions are specified.

The system receives approval.

But who verifies that these measures were implemented correctly?

A requirement written into an assessment does not establish that a corresponding operational control exists.

Ask: Can we demonstrate that every material control required during assessment was implemented before deployment?

Human oversight is among the most frequently prescribed AI risk controls.

Yet its effectiveness depends on operational realities.

Consider an employee reviewing AI-generated recommendations.

Does the employee understand the system’s limitations?

Can they recognize when an output is unreliable?

Do they have sufficient information and time to challenge the recommendation?

Can they override the system without creating operational consequences?

If the answer to these questions is no, human oversight may exist in the governance documentation without functioning as a meaningful safeguard.

Ask: Have we tested whether human reviewers can actually detect and correct AI failures?

An AI system may pass functional testing while still presenting unacceptable risks.

For example, a customer service AI agent may demonstrate strong accuracy on common customer questions.

But what happens when customers provide incomplete information, ask ambiguous questions, or attempt to manipulate the system?

Does the agent recognize when it should escalate?

Does it reliably enforce restrictions on the actions it can perform?

Does it avoid producing confident but unsupported answers?

Functional performance and control effectiveness are related, but they are not interchangeable.

Ask: Are we testing whether the AI performs its intended function, or whether its safeguards work when something goes wrong?

AI systems operate in changing environments.

Models are updated. Data changes. Vendors introduce new capabilities. Employees discover alternative ways to use systems.

A control that worked during initial testing may become ineffective as these conditions evolve.

For example, an escalation mechanism designed for one model version may not perform as expected after an update changes how the system interprets customer requests.

That does not mean every modification requires a complete governance reassessment.

It means organizations need clear criteria for determining when previously validated controls require additional testing.

Ask: What changes would cause us to question whether an existing control still works?

Most enterprises can report how many AI systems have completed governance assessments.

They can track approval times, documentation completeness, and the percentage of systems assigned risk classifications.

These metrics are useful for managing governance operations.

But they do not demonstrate whether controls are functioning effectively.

An organization could report 100% governance compliance while critical controls remain inadequately tested.

Ask: What evidence do we have that our governance controls are reducing the risks they were designed to address?

The solution is not necessarily another governance framework or additional documentation.

Organizations need to connect governance requirements to operational evidence.

Every material AI control should have a clearly defined objective.

If human oversight is required, specify what human reviewers must be able to detect, challenge, or override.

If an AI agent must escalate certain requests, define the conditions under which escalation is mandatory.

If monitoring is required, establish which signals indicate potentially unacceptable performance.

A control that lacks measurable acceptance criteria is difficult to validate and even harder to monitor.

For material risks, organizations should require evidence that controls have been implemented and tested.

Depending on the system, that evidence might include:

Evidence requirements should be proportionate to the system’s risk.

A low-risk internal productivity tool should not face the same verification burden as an AI system influencing financial eligibility, healthcare decisions, or other consequential outcomes.

The objective is not more paperwork. It is better evidence supporting governance decisions.

Pre-deployment testing provides evidence about how a system performs under evaluated conditions.

It cannot establish that every control will remain effective indefinitely.

Organizations should identify which controls require ongoing monitoring, what changes trigger additional verification, and who is responsible for responding when performance falls outside acceptable limits.

For example, a material change to an AI model, its data inputs, intended purpose, or operational environment may justify renewed testing.

Governance approval should establish the conditions under which a system may continue operating—not simply authorize its initial deployment.

Traditional governance metrics remain useful, but they should be complemented by measures of control performance.

Consider tracking:

These measures do not directly prove that every AI risk has been eliminated.

But they offer a stronger basis for evaluating governance effectiveness than assessment completion rates alone.

Imagine your CEO asks a straightforward question:

“How do we know our AI governance controls actually work?”

Could your organization produce evidence?

Not a policy stating that human oversight is required.

Not a completed risk assessment indicating that guardrails are in place.

Not an approval record showing that Legal and Risk reviewed the system.

Actual evidence demonstrating that critical controls were tested, deficiencies were addressed, and safeguards continue to perform as intended.

This is where the distinction between governance activity and governance effectiveness becomes clear.

A risk assessment identifies what could go wrong.

A control establishes what should prevent or mitigate it.

Control verification provides evidence that the safeguard works.

Each serves a different purpose.

And an organization that consistently completes the first two without adequately performing the third may have a governance program that looks mature while leaving significant risks unmanaged.

The answer is not to make governance more complicated.

It is to make governance more consequential.

Because the ultimate measure of an AI governance program is not how many systems it approves. It is whether the controls protecting those systems actually work.

Here’s a question worth asking within your organization:

When your AI governance team approves a system, what evidence demonstrates that its required controls actually work?

Are you validating operational effectiveness, or relying primarily on documentation and attestations?

I’d be interested in hearing how organizations are approaching this challenge.

📩 Want to discuss your AI governance program?

Reply directly to this newsletter. Whether you’re developing your governance framework, struggling to operationalize controls, or looking to improve an existing program, I’m happy to exchange perspectives and discuss the challenges you’re facing.

🔁 Know someone responsible for AI governance, risk, or compliance?

Share this article with them. These are exactly the conversations organizations need to be having as AI moves from experimentation into enterprise operations.

🔥 When AI Governance Meets Reality

This is Part 8 of a series exploring what happens when formal AI governance frameworks encounter the complexities of real enterprise environments.

Coming next — Part 9:

💰 How Much Governance Is Enough for a Low-Risk AI Use Case?

Not every AI system requires the same level of scrutiny. But how do organizations apply proportionality without introducing unnecessary risk?

AI Governance Briefing | Practical insights on AI governance, regulation, and enterprise implementation.

Disclaimer: The views expressed in this article are my own and do not necessarily reflect those of my employer or affiliated institutions. This content is provided for general informational and educational purposes only and does not constitute legal, regulatory, or professional advice.