Confidently Stupid: Why Your Coding Agent Makes Things Up, and How to Make It Stop
2026-07-03
The spec that lied to you
I’ll say the impolite part first: a lot of confident advice about coding agents comes from people who are grading the prose, not the work.
Take a typical spec. It looks organized. It has requirements. It has sections. Someone reads it and thinks, this is really well-written. That is where the failure starts. “Well-written” can hide the fact that the machine has no shared context with you.
I watched this happen on a real PRD for authentication. I gave the agent a direct link to the real documentation. I gave it a detailed spec that was already sitting in the repo. It came back with requirements for a different system. Not slightly wrong. Different.
The missing step was boring: read the docs, read the spec, inspect the repo, reconcile the three. A competent person does that almost reflexively. The model does not have reflexes. It has a next move that looks plausible.
You personified it, so you stopped reading
The reason you accept that broken PRD is that you have personified the model. You assume it will read the docs, cross-check the repo, and reconcile the conflict, because that is what a competent person would do. So you do not really read the output. You skim it. It sounds like someone did the work.
Then you build on it. That is the expensive part. The document has become a false map of the system. Every task downstream inherits the lie.
We personify it because the text arrives dressed like human work. Fluent. Formatted. Confident. Polite. It reads like a mind at work, and that is enough to make smart people stop checking.
You have personified the model so hard that you assume it will check its own work. That is what people do. It is not a person.
AppleScript and lucky paths
So what is it? Two mental models help. Neither is flattering.
From a code standpoint, an LLM is AppleScript on steroids. That sounds reductive. It is also useful. It has an enormous library of moves and no reliable sense of which move fits your actual repo. The fluent output hides that mechanism, even from very good engineers.
From a people standpoint, imagine someone who spent ten years reading internet PDFs and never built the scar tissue. Tons of patterns. No intuition. No instinct that something is off. No “wait, let me check that” reflex.
That is the mechanism underneath both models. The transformer has absorbed a massive number of sequences that looked like success in some context. But success in training is not success in your repo. When you and I succeed, we bring failed attempts, tests, scars, and organizational memory with us. The model does not. It makes one straight-line run and declares victory.
Hallucinations aren’t a bug. They’re a feature.
That straight-line run is the hallucination. The path worked somewhere. It did not work here. The system is not malfunctioning; it is doing what it was built to do, applied to the wrong instance. This is why the “god in a box” belief gets expensive fast. The moment you decide the model can infer anything, you stop giving it the only thing it needs: constraints.
The decision space explodes
I’m a Theory of Constraints practitioner, so I look for the constraint that moves everything else. With coding agents, two things keep biting us: the total decision space and stochastic variability.
Decision space is the number of unresolved choices you are asking the model to cross at once. Those choices do not add. They multiply. Six ordered choices is not six. It is six factorial.
# decisions don’t add, they multiply 6 choices = 6! = 1×2×3×4×5×6 = 720 combinations 34 items ≈ 34! ≈ 2.95 × 10^38 combinations # “well-written” just means you can’t count them anymore
Now look at that “well-written” spec again. Thirty-four items, and each item quietly contains multiple choices. The exact count is not the point. The point is that the model is good at a narrow slice of work, and you just handed it a pile of unresolved decisions pretending to be a plan.
And they are nested. Worse, they loop. Take something that looks atomic: “drive to the store.”
task: “drive to the store” └ car keys └ gas in the tank └ a charged battery └ don’t-get-pulled-over: └ a license plate └ valid registration # and to GET registration... └ know what “registration” means └ know where to get it └ meet every requirement # looped dependency
A three-word errand becomes a dependency tree. It branches and loops. To get registration, you first have to know what registration is, where to get it, and what qualifies you for it. You and I stopped seeing that tree years ago. We compressed it into three words and moved on.
That compression is the problem. “Do the dishes” is three words carrying a lot of assumed context. When you feel how much is packed into a simple instruction, you can finally see the real size of the task you have been forcing down the model’s throat.
The language-shaped promise
There is a second version of this problem. It shows up the moment you let a model write code to do the work, the pattern people now call Code Mode. A working note circulated to the BAML team on Code Mode language surfaces (2026-07-02) named the bug cleanly: Code Mode creates a “language-shaped promise” that is larger than the runtime.
When the model sees “JavaScript,” it sees syntax plus an ambient ecosystem. It writes code that is valid in some JavaScript environment, just not the one you gave it. The trap moves when you switch languages:
The note’s point is that this is not really a syntax problem. The model sees a language name and assumes a library that the restricted runtime may not expose. Standard-looking output becomes hallucinated capability. A Python subset avoids Node-versus-browser confusion, then inherits the same disease: now the model assumes the whole standard library, CPython behavior, and packages it has seen before.
The sharper problem appears when the language is genuinely new. A typed surface can draw a clean, auditable boundary around side effects, which is exactly what you want. But if its standard library is small and new, the model has less pretraining on its names and idioms. With Python or TypeScript, the model already knows a thousand standard patterns. With a fresh surface, even a good one, it has to be taught the boundary. The open question is the right one: what should model-written code be allowed to do without declaring a tool, and what must cross an explicit capability boundary?
Now connect that back to my authentication PRD. The failure is the same shape. The agent never used the Clerk CLI (Clerk is our auth provider). Because it did not call the CLI, it had no way to instantiate something correct and check it against reality. When a model cannot fetch the truth, it invents it: it decides what is correct, then decides the standard for correctness. Error compounds on error.
When a model can’t fetch the truth, it invents it. It decides what’s correct, and it decides the standard for correct.
This is where Code Mode earns its keep, but only when you point it at the right surface. Tell the model: use the Clerk CLI, and read how our organization is actually configured. Now, instead of deciding what “organization” or “authorization” means, it writes a small script that goes and gets the answer.
# BEFORE: the model is the source of truth decide(“what is correct?”) # invented decide(“the standard for correct”) # invented ⇒ error compounding on error # AFTER: the model fetches the truth at runtime run(“clerk ...”) # one real value read(“how is our org configured?”) # one real value ⇒ the decision space collapses
The decision space collapses because the model is no longer guessing across a factorial of possible meanings. It is reading one real value and moving on. Fewer decisions. Fewer lucky paths. Fewer hallucinations.
Three moves that work
The fix is boring, which is why people skip it. Three moves.
- Stop assuming it can infer your environment. Every unstated fact about your repo, conventions, and infrastructure is a decision you silently handed to a straight-line guesser. Say the quiet parts out loud, then give it a way to look up the rest.
- Give it more architectural guidance than you feel it needs. Your feeling is calibrated for a competent human who fills gaps with judgment. The model fills gaps with lucky paths. Over-specify the boundaries, interfaces, and “go look here first” instructions. Guidance that would feel condescending to a person is exactly right for AppleScript on steroids.
- Start from the intention of the code, write tests to that intention, and make the model write the tests first. Intention is the one thing the model cannot infer and the first thing a factorial of decisions buries. A test is intention made executable. Tests-first constrains the work to “make this specific thing true,” and gives the model a checkpoint it would otherwise invent.
Slower is faster
I do not measure agent speed by how fast the first patch appears. I measure total speed across the job. The thing that destroys total speed is rework: work you did, then had to undo, then had to redo. It is the tax you pay for letting a plausible answer become the plan.
All three moves feel slower up front. More specification. More constraints. More tests before any “real” code gets written. Do them anyway. You are trading a little front-loaded effort for less compounding error later. With coding agents, that trade is almost always worth it.
The model is not going to get less confident. That is baked in. The job was never to make it humble. The job is to stop treating a confident, stupid, straight-line guesser like a mind, and to give it an environment so constrained that even a lucky-path machine has trouble missing.
This essay expands a talk from my 2026-07-02 livestream. The Code Mode and language-surface section summarizes and responds to a working note circulated to the BAML team on Code Mode language surfaces and standard-library design (2026-07-02); the examples, framing, and conclusions here are my own. “Code Mode” means having a model write and run code to orchestrate tools rather than calling them directly. BAML is BoundaryML’s typed language for structured LLM output. Clerk is the authentication provider referenced in the worked example.