A previous post explored a scenario where a company uses an AI agent to do a fairly simple task (like reviewing loan applications).
Recap
Assume the AI makes some mistakes, yet we don’t know exactly how often. The company reviews the AI’s decisions, but human effort isn’t set-it-and-forget-it — we respond to incentives. So management offers bonuses for every AI mistake caught. As a result, over time the company ends up in either:
Employees maintain full effort; they eventually learn the AI agent’s actual error rate; or
Employee effort collapses to zero; they eventually believe a false AI error rate, resulting in unexplained losses.
Assumptions are fairly conservative and informed by real experiments, including actual corporate experience.
….with the exception of the AI’s accuracy we which treated a fixed error rate.
Now we allow the accuracy rate to improve. Which is fairly reasonable given the advancements in reinforcement-learning (RL).
To wit:
Over the last year, many such evaluations have found that agents are now able to make progress on tasks where success is easily verifiable, prompting speculation that we are on the verge of RSI.
The claim by Sayash Kapoor and Arvind Narayanan, part of a fascinating post on experiments in recursive self-improvement (“the automation of AI research using AI agents”), question-begs whether RL is sufficiently effective (trustworthy?) to move on to abstract, open-ended tasks.1 This model assumes a simple repetitive task, fully verifiable, and elsewhere Kobe Yank-Jacobs plainly states the obvious: “as things stand, any work given to AI will require human review to make sure it’s up to organizational standards.”
This question — how ought an organization exploit agentic AI — is operative and underexplored. The task remains the same (simple, verifiable) but now the AI model improves in one of two ways:
fixed rate over time; or
variable based on human feedback
Self-Limited RL Improvement
as the AI system approaches “perfection”
→ human confidence increases
→ oversight effort rationally decreases
→ AI appears to continue improving
→ your results are actually getting worse
→ [???]
full effort and accurate assessment of the AI model
OR
zero effort and full confidence in an inaccurate assessment.
→ Losses pile up and you have no idea why
Solutions (Bottom Line)
The problems remain the same (explained below) but also point to fixes:
ballooning bonuses for successful reviews
constantly rotating review teams
They are not costless though.
Graphic Results
Formal Model (Extension-1): The AI Keeps Getting Better
Narrative Description
A fixed error rate is unrealistic. Many of these systems do improve over time, whether through retraining, better data, or just model version iteration. So what if the AI is actually getting better, on its own schedule, independent of anything the analyst or management does?
Here we add a separate force pulling in the opposite direction of human attention decay: the AI’s true error rate falls exogenously.
Formal Description
Parameters (existing)
εt - the AI model’s true error rate, directly unobservable to management and analysts
ε̂t - belief of εt, equal to prior period’s observed error detection rate
L - fixed loss to management per undetected error
γ - analyst’s convex cost of effort
π - bonus to analyst for every caught error
vt - effort (vigilance) level; from non-zero to one. The probability an error is caught
δt - the analyst’s decay parameter, getting smaller over time
δtvt* ≡ effective effort level
rt - the threshold condition (where things go south or not)
Parameter (new)
ρ ∈ (0,1) - AI model’s given improvement rate term
The AI model’s actual error rate process is now defined as:
εt = ε0ρt
The recursion combines this with analyst attention decay (rewriting with k ≡ π*/γ):
Result. Effort still tracks error belief up to a point: vt = min(1, kε̂t-1). Define the analyst effort threshold as ε̄ ≡ 1/k = 2γ/L (determined by analyst incentive bonus):
When true error εt > ε̄ effort is maximized;
When true error εt < ε̄ — when the error rate falls low enough (gets good enough) — maximum effort is not worthwhile for the analyst (the chance of bonus payment isn’t good enough).
Because ε0 (initial error rate) and ρ (improvement rate) are stipulated and fixed terms, the point at which this threshold gets crossed is calculable:
After that point in time, belief declines → effort declines, and attention decay compounds the descent.
Implication. From the outside you can’t tell the difference between an AI that merits less scrutiny (saving time) and an analyst whose vigilance has quietly faded on its own. Override rates and time-per-case are indistinguishable. A company doing everything right and one being fooled look exactly the same.2
To be fair, despite our model setup, the model’s true starting error rate and precise rate of improvement may not be practicably knowable. But the point is these regimes exist and with sufficient data/information a company can characterize the inflection point where you cross from one to the other.
A company winding down its review because the AI earned it, and one winding down because reviews are getting soft, produce exactly the same numbers.
Formal Model (Extension2): When Effort Teaches the AI
Narrative Description
Now the AI agent, still with an initial error rate, improves through human correction. Every caught error is the thing that makes it better. Human feedback remember is subject to endogenous effort. Obviously more errors caught → more improvement. But finding them takes work, and more work as the AI improves.
Formal Description
New Parameter
λ ∈ (0,1) - how effective the AI model improves from identified errors by humans
The model’s true error rate is determined by a combination of the analyst’s vigilance level in the previous period (how likely they are to catch an error) and our new term:
Three things now determine each other jointly, period over period:
Vigilance (effort) ← prior belief of the model’s actual error rate
Belief ← current true error, the analyst’s attention decay, and effort
True error ← prior true error, analyst effort, and the model’s feedback improvement efficiency
There’s no closed-form solution since we need to solve a recursion with three conditions. However, as shown in the graphics above, with some chosen parameters we can display the dynamics over time.
Result. Compounding effects:
Attention decay suppresses actual effort below what the analyst believes they’re delivering (same as earlier model where the AI doesn’t improve)
→ lower realized effort now affects the AI model’s actual performance improvement
→ slower improvement (fewer errors caught means fewer opportunities to feedback the system)
→ lower ceiling on what belief can ever converge to, even if effort later recovers
→ true error always > zero (always some positive error residual)
Attention decay does not just slow this process down. It leaves the system stuck at a strictly worse permanent error rate than the no-decay case would have reached, because the window for real improvement closes sooner.
Implication. In every version of this model built so far, the human side’s failure was a failure of measurement, a belief drifting away from an unchanging or a moving truth. Analyst fatigue doesn’t just lose track of how good the AI is. Given enough time, they can be the reason it never gets as good as it could have.
Memo to management. There are some potential solutions here, but some underlying limitations.
Necessary information:
model’s true starting error rate. Whatever the consultant/vendor says, the stochastic nature means the best we can do is probabilities, and maybe widely dispersed ones at that
company’s loss per error. If the task is simple enough, this can be contained to a definable set. But companies often don’t have this info on hand as much as you’d think.
Solution:
increasing bonuses. Incentive has to scale to compensate for the belief that errors are harder to find.
rotating reviews. It’s easy enough to measure how long people are spending on reviews (though employees hate it). But acknowledging attention decay is pretty universal, so keep teams rotating pretty often to cutoff fatigue earlier. However this imposes some additional organizational cost.
The author is nonresident senior fellow at FAI
One of the eye-openers is that the experiment subjects (frontier agentic AI models) actually didn’t use their entire compute budget!
[R]uns ended with less than 50% of the API budget spent and with hours left before the deadline, even though the agents could monitor their usage and were encouraged to spend down their budgets.
Landing on a principal-agent style model setup, I quickly moved past the idea that the AI model was the “agent” because they have no incentive to shirk. But maybe they do!!! As I understand it, lots of models (not just these) are built with efficiency functions, such that they try to optimize over compute/time when possible, which is a very understandable feature.
A genuinely improving model and a quietly fatiguing reviewer don’t just each explain declining oversight — combined, they hide each other. The model’s real progress makes the analyst’s fading vigilance look like appropriate, earned confidence rather than a lapse.