Who approved this decision?
Maybe you saw that recent story where a bank deployed an AI agent?
“…[It] approved a $1.4 million commercial line of credit. No human reviewed it. No human knew it had happened…nobody was sure exactly who was accountable.
Six weeks later, the agent was rolled back to a lower autonomy tier. Not for a model reason…The decision was operational: The people supervising the agent had stopped looking at the screen.”
There’s a self-limiting paradox at play in agentic AI quality, fed by a natural human tendency. When a company launches an AI system, the general cycle goes:
Human oversight begins diligently (“management says carefully review every case”).
Periodic reports show increasing accuracy across analyst reviews. Override rates starting at 14% go to 8% eventually migrating all the way to 2% after a few months.
The company feels increasingly confident in the system’s accuracy, with good reason! They become more productive by speeding up reviews.
PROFITS!
But our analysts spend less and less time on each review, for two reasons:
Statistically, overrides only occur about 1-in-50 cases. Fairly infrequent. Plus experience has taught them where to look for mistakes. “This must be how we exploit productivity gains from AI,” she thinks. “It’s supposed to make us quicker.”
Productivity aside, humans suffer attention decay.1 Eventually tasks like looking for a rare occurrence results in declining diligence. What psychologists call “vigilance decrement.”
It turns out our mistake rate only looks like it dropped to 2% because HUMANS missed more of them. Type II errors increased. The false sense of improvement combined with our own tendency made us worse off.
Self-Limited Recursive Improvement
as the AI system approaches “perfection”
→ human confidence increases
→ oversight effort rationally decreases
→ AI appears to continue improving
→ your results are actually getting worse
→ [???]
That’s all before adding in task fatigue. I wanted to see if I could model this and extrapolate out where (if anywhere) it ends up.
Conductor: “This is Bottom Line. Last stop before Math-town!”
RESULT: We find you’re basically either in “hell yah!” or “dear god!” territory. Through no fault of anyone, you go speeding toward:
full effort and accurate assessment of the AI model
OR
zero effort and full confidence in an inaccurate assessment.
→ Losses pile up and you have no idea why
Graphic Results
Formal Model
Narrative Description
The intuition is we have a principal-agent problem (literally). At first I thought it was between the analyst and the AI agent, but realized it’s the same old manager-employee. The AI agent is mechanical. Putting aside operating costs, the agent isn’t subject to fatigue, or incentives to shirk.2 Its productivity is entirely a function of its internal design quality and human review effort.
We imagine a company like above. The agent automatically processes cases (applications) which analysts subject to review. Per the story, companies control the frequency of reviews but analysts choose the intensity: total time on each review.
Reviews are not costless, nor are errors.
ANALYST: bears a cost based on how diligently they review, but they also get some payoff from every error caught (a bonus).
MANAGEMENT: bears a fixed cost for each error that gets missed. They choose the size of bonus for actual detected errors.
AI AGENT: doesn’t choose anything, but it’s true accuracy at any time is unknown to humans, who can only make estimates based on observations (i.e., the more you review the more you know).
THIS SETUP ALSO FULLY MIRRORS A BUG BOUNTY PROGRAM, LIKE A FIRM CONTRACTING WITH WHITE HAT HACKERS USING AN AI AGENT.
Formal Description
Parameters
ε - the AI model’s true error rate (fixed), directly unobservable to management and analysts
ε̂t - belief of ε, equal to prior period’s observed error detection rate
L - fixed loss to management per undetected error
γ - analyst’s convex cost of effort
π - bonus to analyst for every caught error
v - effort (vigilance) level; from non-zero to one. The probability an error is caught
Analyst problem. The analyst chooses diligence v ∈ (0,1] to maximize:
yielding an optimum effort v*:
In equilibrium, effort follows beliefs about the model’s error rate. As belief about AI error rate ⬆️ → effort ⬆️
Management problem. Management's expected cost per case is made up of bonuses plus loss from undetected errors:
Substituting the analyst’s optimal response effort, yields in equilibrium an optimal bonus level:
Bonus level is purely a function of how big the loss from an error is, nothing else.
Error rate belief is based on prior period’s observed error rate, which is itself a function of the true error rate and review effort level.
Substituting the analyst's optimal response v* along with the optimal bonus expression π* gives a first-order recursion:
Result. There are two possible outcomes and they are both corner solutions.
Good: When the cost of effort γ is sufficiently cheap, and the loss from errors L sufficiently large such that r > 1, analysts converge to maximum effort and belief of the AI model’s accuracy moves to the actual accuracy (ε̂t = ε), whatever that may happen to be.
Bad: When effort is costly relative to loss from errors such that r < 1, analyst effort collapses to 0 very quickly in response to believing the model has a higher accuracy. It quickly collapses toward 0 (ε̂t → 0).3
Implication. No stable equilibrium exists in-between. It doesn’t matter where your starting assumptions are. The software’s vendor may sell you on some amazing accuracy, or the skeptical manager could declare “this thing is definitely 100% wrong,” at the outset. In the bad outcome, you get a few periods of informative data and after that you’re cooked. But in the good outcome it’s the same thing but the other way and you end up at a stable productive place.
Memo for Management.
For a real-world case, the loss term L is not something easily manipulated by anyone. An AI makes a bad loan decision, it was randomly drawn from the pile — they can’t ahead of time pull a lever to change the cost to the bank. However, somewhat unintuitively, they could limit the AI’s scope to just the highest dollar loans, increasing that pool’s average cost which increases the optimal bonus. That probably sounds scary to a typical bank management.
The other part is the “cost” of detection. Fortunately, this is a reducible factor. Models generally are becoming more user-friendly, more intuitive, more natural language fluent. Also adjacent processes are not fixed. Businesses like our hypothetical bank are built on process improvements: workflows, data management, employee training, etc. Things they invest in all the time.
Ultimately it also depends on the actual quality of the model, there too they're overall improving. But quality is just not fully knowable ex ante. You’re left relying on someone else’s evaluation, and in a world without standardization to boot.
Formal Model (Extension): Attention Decay
Narrative Description
The setup is the same, except now we account for attention decay. Looking for a needle in a haystack gets tiring after awhile. Even a well-meaning analyst exerts the optimal amount of effort per their incentives, but unknowingly their vigilance becomes less effective over time.
You can think of effort v as representing the number of minutes they spend on each review, but those minutes are less effective the more they do. And they don’t realize it.
Formal Description
New Parameters
δt ∈ (0,1] - the decay parameter, declining over time; multiplied by effort level
δtvt* ≡ effective effort level
rt - the threshold condition as before
The analyst continues to solve for vt* (now time-varying to accommodate decay) that maximizes their payoff expression, but they do not observe this time-decaying effort δtvt* is what’s actually being applied.
The recursion condition becomes:
Result. The possible results are same as before, with good and bad edge equilibriums, but now the threshold is moving.
It’s still defined at rt ≶ 1. But since δt is declining, temporarily being in the good space (rt > 1 ) can flip into the bad one (rt < 1). Even without anything else changing. And once crossing into the bad-outcome space, the collapse is geometric and never recovers.
Two analysts, one with and one without the decay condition, would be indistinguishable, to themselves and others.
The author is nonresident senior fellow at FAI
AI models too, but by design. A method to improve efficiency of inference in transformer models is programming a mathematical “decay” in weights assigned to further back tokens in an ongoing context window. IOW the longer your chat goes, it automatically deemphasizes (probabilistically) the earlier prompts. This makes inference faster and lightens compute load. A simple rule that decreases the weight by 1% with each output isn’t very useful, so developers have arrived at self-adjusting decays that respond to the specific prompt sequencing.
Then there’s a related form which describes our hypothetical company, which is when human attention decay meets transformer model reinforcement learning.
A plausible extension would reflect that systems programmed to self-optimize efficiency of compute and speed is akin to worker shirking. In the basic models, a worker wants to work as little as possible - exert minimal effort - without falling below the level where they get fired. The model similarly is trying to minimize “effort.” Of course a worker responds to incentives through pay raises, disciplining, etc. What’s the AI system equivalent?
Never fully quite reaches 0 but asymptotically approaches it. The idea is that my analysts’ effort level is a function of belief of error rate—the likelihood of actually finding a mistake and getting a bonus, if it’s so low they’ll almost never get paid. And that belief is only informed by prior observations. At some threshold little to no observations (effort-adjusted error identifications) are worth it and belief about the model is informed by less and less effort, mistakenly leading to thinking it’s more and more accurate.