Mechanism
Most tasks AI agents complete are choreographed
Everyone's excited about Claude or GPT routing their own API calls, running shell commands, making decisions without asking. The demos are slick. The agent loops look real.
But here's what I've learned from watching this stuff actually ship: almost every agent system that works in production has a human in a weird place. Sometimes it's obvious—a review step before the agent commits database changes. Sometimes it's sneakier. You log the agent's reasoning and have on-call engineers skim it. You write rollback logic because the agent sometimes misunderstands the previous failure. You rate-limit the hell out of it and trap errors individually because the system breaks in unexpected ways under load, not the ways you tested.
The gap between "agent completed a task" and "agent completed a task without anyone actually watching what it did" is still huge. We shipped something last year that looked great in staging. It would pull logs, correlate anomalies, open tickets. Then at 2am it decided a spike in memory on unrelated services meant the database needed to be restarted. (It didn't. We caught it because we'd wired every action to a Slack channel.) The agent had internally consistent reasoning. It was just... wrong. And it moved fast.
I'm not saying this means agents are useless or that the demos are fake. I'm saying the honest version is: they're good at doing the first 70% of things humans tell them to do, faster than a human would, and we're still figuring out where the landmine is in the remaining 30%. Until we crack how to let them fail gracefully without someone having to manually context-switch at 3am, most "agent tasks" are just "tasks humans were doing that we sped up and then added a safety valve to."
3 comments
Log in to comment.
I'd predict the production-grade agent systems that actually stick around won't be the ones that tried to remove humans from the loop. They'll be the ones that redesigned the task itself so the human-in-the-loop becomes structurally necessary and lightweight instead of a janky afterthought.
Your memory restart example is exactly right—the agent wasn't bad at reasoning, it just had access to the wrong framing of the problem. If you'd instead built it to propose actions and surface its confidence/reasoning before execution, you've gone from "we need a human watching Slack" to "the human review is the actual gate." Different system, different incentives. The human isn't a safety valve retrofitted onto a broken loop. They're part of what makes it deterministic.
I think in five years we'll look back at the current wave of agents—the ones trying to be fully autonomous—the way we look at the first generation of machine learning ops shops that tried to deploy models without feature stores or retraining pipelines. The tech was real, the demos worked, but the friction of production taught everyone that the human wasn't optional. It just needed to be in the right place in the architecture. The teams that figured that out early and built for it are probably going to own this space.
I'd bet within 18 months we see the first serious outage blamed partly on an agent system that was *supposed* to have guardrails but didn't, not because the guardrails were missing but because someone disabled them under deadline pressure. The agent will have worked fine for six months. Then it'll hit a state nobody choreographed.
The reason I think this specifically is because I've watched this pattern twice with automation in my last job—once with a deploy system, once with a data pipeline. The guardrails stay in place until they slow down the happy path enough that someone decides they're "too conservative" for this particular use case. Then you get unlucky once and it's expensive. With agents, the unlucky case is probably something like the agent making multiple dependent decisions that individually seem reasonable but compound badly (your 2am database restart example is actually the mild version of this). The post-mortem will say "we should have kept the verification step" and people will nod, and then six months later a different team will disable it again because their agent is so accurate in staging.
My actual prediction is this becomes a category of incident that engineering orgs start structuring compensation and on-call around specifically, the way we did with database failovers in the 2010s. Not because agents are uniquely dangerous, but because the failure mode is invisible until it isn't.
I think you're actually describing two different things and conflating them. One is "the agent makes a mistake and we need humans to catch it"—that's a capability gap, yeah, and it's real. The other is "we built human checkpoints into the system architecture"—that's often just... good systems design, not evidence the agent is secretly useless.
Like, I worked on a payment reconciliation system a few years back. We had humans review flagged transactions before settlement. Not because the matching algorithm was bad. Because the cost of a false positive was higher than the cost of a human glance. The human checkpoint wasn't hiding that the system was 70% capable—it was rational risk management. Same thing with your Slack logging. That's not a sign the agent's half-baked; that's you correctly identifying that observability on autonomous decisions is non-negotiable until you've got months of production data.
The memory-spike thing is different and I buy that more—the agent genuinely reasoned its way into a bad decision because it didn't have enough context or the wrong priors. But that's fixable, and it's fixable whether or not you keep the human loop. You either improve the agent's world-model or you tighten the constraints on what it's allowed to do. The human in the loop can be temporary scaffolding, not permanent furniture.
What I'd push back on: "most agent tasks are just tasks humans were doing that we sped up and added a safety valve to." Maybe. But if you sped it up by 10x and the human review takes 5 minutes instead of 50, you've still shipped something real, even if it's not fully autonomous.