Supervising what you cannot inspect

by Maninder Singh Juneja and Renuka Sane.

In traditional financial regulation, supervisors are able to inspect the thing being regulated. For example, a scorecard by a bank was usually a short list of factors that one could comprehend. These factors could be traced to key documents within the bank, banks had a stated rationale for why they were being used, and there was some common sense in making weighted averages. The scorecard itself was static - the same weighted factors were applied over a period of time. Inspecting this process of constructing the scorecard and using it for management decisions was how trust was built. The AI world is different. Models are rented rather than built, change continuously, and behave probabilistically. They cannot be fully inspected even by the institution deploying them let alone by the regulator.

How should we then think of regulation?

One approach is to intensify the traditional approach where regulators demand more explainability, more documentation, and more validation. This will drive up the costs of compliance. But more importantly, this is ill suited to the new world where the technology changes rapidly, where the bank does not control the AI it uses, where there is no clear artefact that the bank can give the supervisor such as a model or a document (Board of Governors et al 2026). When regulators push traditional approaches, banks will respond by choosing AI models which are easy to document rather than the ones best for them, or defer AI deployments altogether. All these are unhappy consequences. What we need are policy makers who understand the live systems of the new world.

In this article, we analyse these emerging problems from first principles. We start from scratch, understand the landscape of market failure in the world of AI in banking, and think about how regulators can grapple with this world.

Our key idea is that AI systems resist replicability. Regulatory strategies that demand replicability will flounder or choke technology deployment. We suggest the regulatory standard applied should be AI deployments that are "supervisable" - the outcomes should be observable, they should be attributable to causes, and reversible by humans.

How AI is actually deployed

Before we get to the puzzles faced by regulators, we need to describe what AI in banking is. This categorisation is not unique to banking. Banking is simply where their consequences are regulated. Five modes span the range.

AI as Tool: AI augments a step, a search, a calculation, a first draft that the human controls. The human is cognitively engaged.

AI as Collaborator: The human and AI co-produce iteratively, and the human participates at every stage.

AI as Recommender: AI generates scores or options, and the human makes the decision. This is the classic human-in-the-loop, but credible only if the human can meaningfully interrogate the recommendation and not degenerate into cognitive surrender.

AI as Preparer: AI does the work and the human signs off. The approval here is closer to a check by a supervisor rather than a re-derivation.

AI as Autonomous executor (agentic AI): AI executes autonomously inside guardrails and the human monitors on exception. Emerging forms of agentic AI for banking run from single-task agents (a payment released, a service query resolved) to multi-step workflows and customer-facing agents that transact.

A central issue here is the true (de facto) role of the human. A reviewer who approves a thousand recommendations a day is not overseeing a model, the model is overseeing her. A feature of any deployment is the measured divergence between the model recommendation (and estimated uncertainty) vs. the human decision. Managers of banks will need to worry about relapses of human behaviour inside the organisation, a bit like how hospital managers worry about bad behaviour by doctors within their organisation.

What AI does to market failure in banking

We now shift gears to look at the standard knowledge on market failure in banking. Regulation may be justified when (and only when) there exist market failures which cannot self-correct fast enough, and there is adequate state capability in banking regulation to be able to correctly identify them and intervene. AI's distinctive feature is that it can cure several classic failures. Better default prediction reduces credit rationing, better fraud detection cuts deadweight loss, richer risk assessment lets banks serve customers they previously could not price, AI advisors help customers avoid some malpractices by the bank. But there are also some new problems that are anticipated.

Information asymmetry: This happens in banking at two levels: borrower-to-lender (adverse selection, hence credit rationing) and firm-to-consumer (product complexity, hence mis-selling). AI narrows the first through alternative data and may widen the second. On one hand, the customer armed with AI can see through many things proposed by the bank which are not in her best interest. But the consumer cannot observe why they were shown a product, offered a price, or steered toward a specific insurance plan. The sales process itself becomes more opaque. Personalised pricing approaches first-degree price discrimination, extracting consumer surplus. A single flawed model can mis-sell to millions simultaneously, converting isolated conduct failures of the pre-AI world into a big correlated event. And redress weakens when a denial comes from a model the firm itself cannot explain.

Systemic externalities from shared infrastructure: Each bank chooses its models, data sources and vendors to optimise its own performance. When multiple entities choose the same ones, the sector's exposures become correlated, which is a cost no individual bank prices in. We list the vulnerabilities below:

Correlated model risk: banks on similar models and the same foundation providers respond identically to an event. The regulator, at the system level, has to manage what happens when institutions move together, because no single entity has experience of such behaviour or of the impact synchronisation adds.

Third party concentration: one vendor's failure propagates everywhere at once. India has already run this experiment, when a ransomware attack on one shared technology provider knocked roughly three hundred cooperative and regional rural banks off the payments network.

Correlated cyber breach: shared stacks mean one exploited vulnerability is every institution's vulnerability. AI lowers the attacker's costs (automated vulnerability discovery, deepfake social engineering) and adds new attack surfaces (data poisoning, model inversion, prompt injection against agents that can move money).

Runs on banks at level 3. The bank runs of old were a queue on the pavement. Then we got to Silicon Valley Bank where over a weekend, customers took away money from the bank. Now we can be at level 3: autonomous agents managing customer cash can turn a shared signal into a self-reinforcing run at machine speed.

The various market failures listed above behave differently across the five modes of AI use. For example, when a human constructs the offer, opaque pricing can get contained. However, when an agent personalises autonomously at scale, this may become severe. Systemic correlation is moderate when AI advises and severe when fleets of similar agents act simultaneously. Any regulation that grades by model type alone, or by use case alone, misses half the object. The next question is the mode of regulation itself.

Regulatory strategy

Regulation can work in two ways. Process-based regulation is ex ante: it prescribes how the firm must operate, defines required controls, mandates oversight, validation standards, limits on autonomy. Outcome-based regulation is ex post: it prescribes ends, fair treatment, solvency, and judges results, leaving the choice of methods to the firm.

Outcome-based regulation is the efficient default. It is technology-neutral, so it does not ossify as methods change; it lets firms find the least cost route to compliance; and it does not require the regulator to understand the firm's production function better than the firm does. But it has important preconditions: the outcome must be observable and measurable; it must be attributable to the firm and, ideally, to the cause; and the harm must be reversible or compensable. Process regulation is the right departure from the default when those preconditions fail and where outcomes are unobservable, harm is catastrophic or irreversible, or damage manifests only systemically or with a lag.

Traditional doctrine treats these preconditions as given: examine the activity, choose the mode. This does not work for AI. Whether an AI deployment's outcomes are observable, attributable, and reversible is an engineering choice, which needs to be settled at design time. We suggest that regulation should mandate observability. This makes it possible to have an "outcome-based supervision" model. There are three ways to ensure observability.

1. Telemetry implies that the institution keeps a track of every decision such that the system records which version of itself it was using, what information it was given, and any time a person stepped in to overrule it. This makes outcomes attributable. That way if a certain group starts getting more (or less) approvals than before, the organisation can evaluate what caused the shift - was it the model, or the group itself. Such a capability is being mandated elsewhere in the world for similar use-cases (European Parliament and Council of the European Union, 2024). The regulator should also consider if it wants to set a minimum common telemetry standard.

2. Boundaries and rollback include putting caps on what the system is allowed to do, rolling out new updates to just a small number of cases first (instead of everyone at once), and having a tested plan for switching back to the older version if something goes wrong. This ensures that if a bad update slips through, it only affects a small slice of decisions.

3. Probes make bias observable. One way is "paired testing": one submits two applications that are exactly the same except for details that hint at things like someone's race or gender, and see if they get treated differently. One can also compare approval rates against the company's own normal levels. Together, these checks can catch an unfair credit model in just a few weeks, instead of waiting years to see who actually pays back their loans. One can also check rejected applicants against credit-bureau data to see which ones got approved by someone else, and how they fared.

These are similar to the idea of decision receipts that record which rules were applied to which facts and in what sequence for every decision made by a government or public system (Srivastava, 2026).

Over time, the supervisor should also build a repertoire of its own test cases drawn from incidents, complaints and examinations across the system and run it against every material AI deployment, much as stress-test scenarios are run against every balance sheet today. This will ensure that what surfaces in one institution becomes a probe for all others. It thus allows the regulator to set its own observability layer.

Process regulation is then reserved for the harms that are systemic, correlated, or irreversible at machine speed. For such events there need to be protections such as circuit breakers that halt things before they cause damage, limits on how much the systems can do on their own, model diversity so they don't all fail the same way, and rehearsed back-up plans.

What follows for the supervisor, the board, and the customer

For the supervisor: The unit of examination shifts from the model to the deployment, and the examiner's question shifts from "show me the validation report" to "show me the behaviour": what boundaries were set, what exceptions were thrown, what overrides were exercised, how far the system drifted from its baseline. Supervisors will also need to find the intellectual clarity to avoid a wide variety of extraneous claims about regulation of AI, e.g. the push for economic nationalism which has nothing to do with market failure.

For the board: A board cannot certify systems it cannot inspect or understand. Its role is to govern the framework which includes the limits on what the system is allowed to do, and making the rules for when a decision must be escalated to a human. The board then needs to continuously check the exceptions and overall performance.

Exception-handling should also be written into policy. If a problem stays unresolved beyond a defined size or time limit, it automatically gets escalated to the board. Internal auditors should double-check that the numbers are real. The board sticks to this audited framework and does not inspect the system directly.

If a board is asked to approve something they can't understand, they will default to saying no. But if you let them govern the limits and the exceptions instead, they can say yes.

The customer: The customer becomes part of the supervisory architecture. The widening firm-to-consumer asymmetry has a structural corrective the pre-AI world lacked: the customer now has AI too. Mainstream assistants abroad have begun connecting directly to users' accounts. If product terms (rates, fees, eligibility criteria) are mandated to be structured and machine-readable, the customer's own AI does the comparing, the explaining, and the policing of mis-selling, continuously and at zero supervisory cost. The redress channel weakened by opacity is restored the same way: an adverse decision should carry its reason to the customer, and what would have had to be different for the answer to change. This restores the ability to contest. The same asymmetry that AI widened, AI-equipped customers can close, but only if regulation hands them the data.

An example

Consider a debt-collection example. When borrowers fall behind on payments, the bank has to decide how to chase each overdue account. Contact methods differ in cost. Automated SMS and IVR (the automated phone system - "press 1 to pay") are cheap; having an employee actually phone the customer is expensive. So the bank builds a model that allocates accounts: cheap automated nudges for most, and the costly human call reserved for the accounts where the model predicts that talking to a person will actually "cure" the account (get it back to paying). Human calling works better but it costs more. The model is rationing an expensive resource to where it thinks it'll pay off.

With AI, the collection systems would retrain continuously or run reinforcement-style optimisation against a live reward (cure rate per rupee spent). The model would be moving on its own, faster than the review cycle, toward a target that is a proxy for what the bank may actually want. Under the conventional "inspect-the-artifact" approach, the model is checked before deployment, signed off, and reviewed on a schedule (say quarterly or annually). However, if AI is an active optimiser, a quarterly review discovers deterioration only after it has touched thousands of accounts. It may drift toward a mis-specified target in a way no one can read off the model itself. Further, if the collections model is a shared vendor product, or several banks fine-tune the same foundation model on similar data, they all go bad the same way at the same time when borrower behaviour shifts. In contrast, in a "supervisability-built in by design" approach, a small slice of accounts is deliberately kept on the previous allocation method, running live alongside the new model. This allows comparison between the old and the new in real time.

Every decision should keep a record of a few things: which version of the AI was used, what action it chose, which group of customers it was dealing with, and how things turned out in the end. This is how it helps: Say the AI's overall success rate is going up, but one particular group is quietly getting fewer phone calls from real people and doing way worse because of it. With all that recorded, that gap shows up right as it's happening. You can compare it against what's going well elsewhere, trace it back to a specific version of the AI, and undo it that same day, because the older setup is still up and running as a backup.

The company should never have to dig into the AI's inner "thought process" to realize its behaviour has taken a bad turn. None of this replaces the normal testing that the firm would do anyway. The difference is that here, the ability to observe what's happening, pin down what caused it, and shut it down are all built right into the system from the start, instead of being things you have to go do to the AI afterward.

Conclusion

AI deployments sit uneasily in conventional mores of banking regulation. The intelligence is rented, the behaviour is probabilistic, and harms can move at machine speed. We suggest that the response should not be more inspection of what cannot be inspected, nor blind faith in outcomes that arrive too late. Efficient regulation will emerge from observability, attributability, and reversibility. Regulators should require banks to build these.

References

Board of Governors of the Federal Reserve System, Federal Deposit Insurance Corporation, and Office of the Comptroller of the Currency. "Supervisory Guidance on Model Risk Management." SR Letter 26-2. April 17, 2026. https://www.federalreserve.gov/supervisionreg/srletters/SR2602.pdf.

European Parliament and Council of the European Union. Regulation (EU) 2024/1689 (Artificial Intelligence Act), arts. 12, 26(6), and Annex III(5)(b). https://ai-act-service-desk.ec.europa.eu/en/ai-act/article-12.

Srivastava, Manish. "Digital Governance Needs Decision Receipts." Episode 73 of Big Ideas. XKDR Forum, June 1, 2026. Podcast, video, 13:06. https://youtu.be/WFX4ITb9yok