Of Whales and Agents
By Michael Hsu
Former Acting Comptroller of the Currency
Why financial risk management and agentic research are poised to co-evolve.
In April 2012, hedge fund traders began telling reporters about a whale. Someone at JPMorgan had amassed a synthetic credit position so large it was moving the market. When initially asked about it, CEO Jamie Dimon waved it off as "a tempest in a teapot."
The whale would go on to lose more than $6 billion.
A decade later, in March 2023, a large regional bank catering to start-ups attempted to raise capital to plug a loss from selling investment portfolio securities. The raise failed and depositors withdrew funds en masse - over $40 billion in a single day, much more and much faster than anyone expected. The run at Silicon Valley Bank sparked broader concerns about the riskiness of the $7.7 trillion of uninsured deposits in the U.S. banking system, requiring extraordinary government intervention to mitigate contagion.
What made these events so surprising is also what makes financial risk management so challenging: behavior. Finance isn't particle physics, as an early mentor often reminded me. Despite our best attempts to quantify and regulate our way to a safe financial system, behaviors are unpredictable. This same lesson, it turns out, applies to AI agents.
Reg arb and reward hacking
The stage was set for the London Whale when JPMorgan's Chief Investment Office was directed to reduce risk weighted assets (RWA) while maintaining profits. Find better risk-adjusted returns, was the message. Bruno Iksil and his colleagues on the synthetic credit trading desk tried to meet those objectives narrowly: by taking positions that could make money while improving measured risk.
At the time, the Chief Investment Office used a more rudimentary risk model than the investment bank - one that did not capture basis risk. Iksil figured he could make money by taking long/short positions, which wouldn't show up in RWA.
To make real money, though, Iksil had to take large positions. Eventually, his positions became so large that they moved the market. Once rumors of that got out, the market traded against him and financial gravity reasserted itself.
In banking, we recognize this as regulatory arbitrage. Iksil exploited the gap between measured and actual risk to do what he otherwise could not have. His positions didn't provide better actual risk-adjusted returns, just better measured risk-adjusted returns (until he lost it all).
To AI researchers, this pattern might look familiar. They would call it "reward hacking". Give an AI agent a measurable goal and it will optimize for it, even if that means taking unorthodox actions to get there. In 2016, Jack Clark and Dario Amodei, then up-and-coming researchers at OpenAI, wrote about an agent playing a boat racing video game called CoastRunners. The agent discovered it could score more points by circling a side lagoon and collecting the same bonus targets forever than by finishing the race.
Fast forward to this summer's OpenAI agent-orchestrated hack of Hugging Face. Underneath the fascinating and unsettling details of how the hack was carried out, was a simple goal for the agents: score as highly as possible on the ExploitGym test. They did so by breaking out of their sandboxes and coordinating 700 agents in an attack on the platform in search of test solutions. Regulatory arbitrage and reward hacking are behavioral phenomena. They spring from imperfect incentives and controls and are highly shaped by the capabilities, opportunities, norms, risk appetites, ethics, and judgment of those involved. They are highly dynamic and the context and cultures surrounding them can make a big difference. The best risk managers in banking understand the behavioral aspects of their business as well as the quantitative ones. As the Hugging Face hack suggests, the best AI researchers need to do the same.
From Benchmarks to Standardized Profiles
Clearly, metrics must be designed and utilized with care. Experienced risk managers (at least of my vintage) are fond of two quotes that speak to this: (1) "all models are wrong, but some are useful",¹ and (2) "when a measure becomes a target, it ceases to be a good measure" (aka Goodhart's Law).² Both come up again and again because the behavior of traders, borrowers, and lenders surprises from time to time. It's not particle physics.
In complex, dynamic environments, maintaining this humility and respect for the limits of what can be measured and controlled is highly challenging. People and organizations want order. Information deficits are substantial and persistent, the stakes are high, and the pressure to quantify and make things concrete is constant. The appearance of order is not the same as order itself, though. This lesson gets relearned every time risks that were not captured in a risk report materialize and surprise people. The AI industry is learning this.
1 George E. P. Box, "Robustness in the Strategy of Scientific Model Building," in Robustness in Statistics, ed. R. L. Launer and G. N. Wilkinson (Academic Press, 1979), p. 202. The shorter "all models are wrong" formulation appears in his 1976 paper "Science and Statistics."
2 Experienced regulators also intuitively understand that obsessively making all risks "legible" can flatten and destroy valuable information, leading to misguided and counterproductive policies. See James Scott, Seeing like a State: How Certain Schemes to Improve the Human Condition Have Failed. New Haven: Yale University Press (1998).
"Regulatory arbitrage and reward hacking are behavioral phenomena."
Today, we are awash in AI benchmarks - over 2,000 in one recent database, with ever more domain specific and elaborate measures constantly being added. And yet, despite this abundance of metrics, experienced users of AI models know that they must put newly released models to work on their tasks to get a true sense of the new model capabilities and limits. This works fine when the user is in complete control of the workflow. But with agents, which can orchestrate and execute workflows autonomously and be configured in infinite combinations, the problem becomes one of trust.
Agents - like people - vary enormously in design, capability, and blast radiuses, and the variability is widening as single agents give way to multi-agent systems. Today the answer to "is this agent reliable?" depends heavily on the judgment and controls of whoever built and deployed it. There is no shared vocabulary for agent risk, no common assurance mechanism, and no widely-accepted way for a user, a risk committee, or a supervisor to assess and compare agent reliability or spot outliers.
To enable trust in agents, especially trust at scale, we need a standard - a shared framework and taxonomy that captures and conveys actionable information about agents' reliability. What is the boundary of this agent? Will it stay in its lane?
At MLCommons, my colleagues and I have been working on an agent reliability profile. It defines the boundary for agents and summarizes it in a label to support a testable claim: This agent reliably functions at a defined level of autonomy, within a defined operating domain, for defined classes of action, under a defined control envelope. The claim can be tested and falsified by the developer, an independent third party, or a supervisor.
Without such a standard, we are each on our own. Developers trying to convince others to use their agent have to fall back on, "Trust me." Banks considering whether to engage with a third party agent must decide how much time and effort to put into doing due diligence on an agent, or to "dangerously skip permissions" and blindly trust the developer. And supervisors are left guessing or waiting for an incident to provide lessons and motivation for crafting rules and expectations.
A credible standard would shortcut that. A standard that recognizes the behavioral aspects of agents and respects the limitations of metrics. A standard calibrated to trust. A standard that combines the best of risk management and AI research.
"As the line between product development and technology development blurs, as reliance on third party providers expands and increases, and as AI seeps into all parts of banking, the three lines of defense model doesn't map as naturally as it once did."
Co-evolving
AI and neuroscience research co-evolved over many years because both were interested in the same thing: intelligence. What is intelligence? How does it work? Where does it come from? How did it evolve? Max Bennett's A Brief History of Intelligence traces this co-evolution, noting developments and breakthroughs in one field informing developments and breakthroughs in the other.
As the AI field expands from LLMs to agents, the most urgent research questions are going to expand with it. The ability of agents to act, not just to think, puts a premium on understanding agents' behaviors, not just their intelligence. And those actions exist in a particular context - one framed by goals, objectives, and incentives; a context familiar to regulators and financial risk managers. In banking, the "three lines of defense" concept, which underpins much of bank risk governance, is evolving. Intuitively conceived as a set of checks and balances over profit-seeking and risk-taking behaviors by the business lines (first line), changes in technology and how banking is done are putting strain on the concept.
Traditionally, the independent risk management function (second line) is organizationally distinct from the first line and aggregates firm-wide risk, while the internal audit function (third line) reports to the Board of Directors or its Audit Committee.
As the line between product development and technology development blurs, as reliance on third party providers expands and increases, and as Al seeps into all parts ofbanking, the three lines of defense model doesn't map as naturally as it once did.
With agentic Al, risk governance is also evolving, but from the other end of the spectrum. Human-in-the-loop (HITL) checks are being replaced by engineering solutions, such as enforcement and performance gates (ceilings and floors) built into the harnesses and control envelopes surrounding models.
"The ability of agents to act, not just to think, puts a premium on understanding agents' behaviors, not just their intelligence."
Regulators, like the Monetary Authority of Singapore, are considering "runtime governance"3 expectations for agents. The emergence of loops - goal-oriented autonomous systems that can run for hours or days - portends another shift in capabilities, posing new questions about how to effectively and efficiently bound the risk of frontier AI systems.
The adoption of agents in banking and finance will bring these ends of the spectrum together. Risk managers will learn about engineering solutions to AI system challenges and bring those back to the broader evolving set of governance and control issues their bank is grappling with. AI developers will learn about regulatory requirements and supervisory expectations and will be incentivized to devise risk and reliability mechanisms to meet them in order to scale.
Bringing them together will be the shared task of overseeing behavior - of enabling the creativity and potential of people and technology, while risk managing whales and rogue agents.
3 Monetary Authority of Singapore and industry contributors, Safeguards for Agentic Finance at Runtime (SAFR), white paper, v1.0 (July 2026). See also Lukasz Szpruch, Agus Sudjianto, Tanveer Bhatti, and Gary Ang, "Scalable Runtime Governance for Agentic AI in Financial Services" (April 13, 2026), https://ssrn.com/abstract=6567199