Why Financial Services Leaders Are Right to Be Nervous About AI Agents

Something interesting happens when you talk to transformation leads and COOs in financial services about agentic AI. They are not dismissive of it. Most of them can see exactly what it could do for their operations. But there is a hesitation that does not go away, even when the technology is working as described and the vendor demos are compelling.

That hesitation is not technophobia. It is professional judgment, and it deserves to be taken seriously rather than managed away.

The question worth asking is not how to overcome nervousness about AI agents. It is what that nervousness is actually pointing to.

The regulatory gap is real, and everyone knows it.

Financial services operates under some of the most detailed regulatory frameworks of any industry. And yet when it comes to agentic AI, the frameworks are genuinely underdeveloped. Regulators are watching, consulting, and publishing guidance notes. They are not, in most cases, providing the kind of clear accountability structures that financial institutions need to deploy with confidence.

This is not a criticism of regulators. The technology is moving faster than any governance framework can reasonably track. But it creates a specific problem for firms that operate in environments where “we followed the rules” is a necessary condition for any deployment decision.

When the rules are unclear, risk-averse organisations face a choice between waiting for clarity that may not arrive on any useful timeline, or deploying and hoping that what they build will be deemed appropriate in retrospect. Neither option is comfortable. Neither should be.

What sits underneath most of the nervousness about agentic AI in financial services is not a failure of imagination about what the technology can do. It is a legitimate concern about who is accountable when an agent does something that causes harm, and whether that question has a satisfactory answer yet.

It largely does not.

The judgment gap is harder to see but more consequential.

Regulatory uncertainty is visible. The judgment gap is less so, and in some ways more consequential.

Agentic AI systems are not rule-executors in the way that earlier automation was. They make decisions. They act across sequences of steps. They handle exceptions and edge cases in ways that were not explicitly programmed. That is what makes them powerful. It is also what makes them difficult to deploy responsibly in high-stakes environments.

The difficulty is this: the judgment that experienced people exercise in financial services workflows was never designed to be made explicit. It accumulated. A compliance officer who has spent fifteen years reviewing suspicious activity reports is drawing on a pattern library that exists nowhere in writing. A credit analyst who can tell from the shape of a business’s cash flows that something does not add up is applying a form of inference that is genuinely hard to articulate, let alone encode.

When an agentic system operates in those same workflows, it is not drawing on that judgment. It is drawing on its training data, its parameters, and the instructions it has been given. That is a different thing. And the gap between the two is not a gap that better technology closes. It is a gap that better design has to account for.

The question for any organisation deploying AI agents into judgment-rich workflows is not whether the system will perform well on average. It is what happens in the cases where it does not, and whether the humans around it are positioned to catch those cases before they become problems.

Trust is a design problem, not a technology problem.

There is a version of the conversation about AI trust that locates the problem in the technology. If the model were more accurate, more explainable, more consistent, then trust would follow. That framing is appealing because it puts the solution in the hands of AI developers and gives institutions a reason to wait.

It is also wrong, or at least incomplete.

Trust in any consequential system, human or automated, is not a function of its technical performance alone. It is a function of how the system is embedded in an organisational context. Who monitors it. What happens when it makes an error. Whether the people working alongside it have meaningful ability to question its outputs. Whether accountability is clear when something goes wrong.

These are design questions. And they apply as much to the deployment architecture as to the model itself.

An AI agent operating in a KYC workflow without clear escalation paths, without a defined audit trail, without a human role that is genuinely positioned to intervene and not just formally present, is a liability regardless of how well the underlying model performs. Conversely, an AI agent operating with thoughtfully designed human oversight, in a workflow that has been mapped for where judgment is required and where it is not, in an organisation where the accountability structure is clear, is a different proposition entirely.

The trust problem is not solved by waiting for better AI. It is solved by building better systems around the AI you are deploying now.

Read about how Experience Haus approaches the Trust dimension of Intelligent Workflow Design.

What a trust-ready deployment actually looks like.

Organisations that are deploying agentic AI responsibly in financial services are not the ones who have resolved all the uncertainty. Nobody has done that. They are the ones who have been deliberate about a small number of things that make the difference.

They know exactly where in a workflow a human is accountable, and they have designed that accountability to be real rather than nominal. A human who is technically in the loop but is processing two hundred AI-generated outputs an hour is not meaningfully exercising oversight. They are a rubber stamp with a salary. That is not a defensible position, and most regulators will see through it quickly.

They have built audit trails that reflect what the system actually did, not just what it was supposed to do. When something goes wrong, they can reconstruct the decision sequence. They can explain it to a regulator, to a client, and to their own board. That explainability is not an afterthought. It was designed in from the start.

They treat the first phase of deployment as an evidence-gathering exercise. The AI operates in assist mode, supporting human decisions rather than replacing them. The organisation watches where the system performs well, where it misses, and where the humans around it are making different calls and why. That evidence informs the next phase of deployment. It is also the basis for a conversation with regulators that is grounded in data rather than assertion.

None of this is complicated in principle. It requires deliberate choices at the point of design, and a willingness to treat the deployment architecture as a serious piece of work rather than an implementation detail.

The nervousness is real.

The nervousness that financial services leaders feel about agentic AI is pointing at something real. The regulatory environment has not caught up. The judgment that experienced people carry in high-stakes workflows is genuinely difficult to replicate or account for. And trust, properly understood, is something that has to be built into how these systems are deployed, not assumed because the technology is performing as expected.

Getting this right is not primarily a technology challenge. It is a design challenge. And it is one that repays serious attention before the first agent goes live, not after.

If you are working through these questions in your organisation and want a structured way to approach them, we would welcome the conversation.