Absent or Outvoted — Why AI Struggles With Enterprise Software
Two failures from real builds show two different reasons AI gets enterprise software wrong: one answer was never written down, the other was — but lost to the volume of outdated material saying something else.
Alice moves next Thursday’s committee meeting into a different room. A few minutes later Dan opens the same event and shifts the start time by half an hour. Both save. One of those changes is gone, and neither of them knows which.
I built that clash myself, on a demo calendar app I put together to test what AI-assisted development is actually good at now. And “now” matters, because the honest answer keeps changing under you. These tools have got properly good, fast, at the parts of enterprise software that show up on a checklist — authentication, audit logs, the wiring that used to eat a week of anyone’s life. Or more precisely, they’ve got good at the versions of that wiring which appear in a thousand tutorials, on the stacks those tutorials use.
Step off that path and one of two things happens. Either the answer you need was never written down, or it was written down and loses to the sheer volume of material saying something else. Absent, or outvoted. This post is one case of each.
In Data Integrity in an Age of Autonomous Writes I argued that the alarming agent failures are the prosaic ones rather than the sophisticated ones — half-finished records and plausible wrong values, not race conditions. I still think that’s where the volume is. But I set the sophisticated cases aside too quickly, and the first half of this post is me going back for one of them.
The one nobody wrote down
The fix for Alice and Dan is old and good. Fowler wrote it up as Optimistic Offline Lock in Patterns of Enterprise Application Architecture, over twenty years ago: check whether the record you’re saving has changed since you read it, and refuse the save if it has. It has held up well.
Notice precisely what it establishes, though. It checks that the client has seen current state. It doesn’t check that what the client is sending was built from it. Those come to the same thing when every caller is a person editing the screen in front of them, because a person who re-reads the screen sees the new value sitting there.
So I ran the thought experiment further. Bring an AI agent into the mix — call him Robert — reaching the calendar over MCP, the protocol that lets a model call tools and services directly. Robert submits a change and it’s rejected: the copy he’s working from is stale. He reads the rejection, works out exactly why, and does the obvious thing. He refreshes the version he’s holding and resubmits the request he’d already built.
Nothing about that is malicious, and nothing about it is broken. Every step is correct. But the body he resubmits was assembled from what he read before Alice touched it, so Alice’s change goes back to its old value underneath his write. Nobody tells her.
Whether this bites you depends on something most people never think about: where the version travels. If it’s embedded in the payload, the system protects itself — resubmitting the body resubmits the stale version, and the second attempt fails too. If it travels alongside the payload, in an If-Match header or as a separate parameter, then “refresh the precondition and resend the entity” is exactly what a naive retry loop will do.
Which is where my own fix turns out to make things worse. I’d built a log of GUIDs issued on every read, so a client can prove it’s working from a current copy of the data. That token is out-of-band by construction — it isn’t domain data, so it can’t live in the payload — and that separation is precisely what makes the naive retry dangerous. The token attests to the read, not to the write. It tells you the client has looked at current state. It says nothing about whether what the client is sending was derived from it.
I haven’t built the next step, but I think there’s a way forward. If the server keeps the snapshot behind each token, the token stops being a freshness proof and becomes a base pointer: diff what arrives against what was read, apply only what actually changed, and Alice keeps her room change whether or not Robert behaved well.
It leaves a residue, though. When Alice and Robert change the same field, both writes are honest, both are current, and one of them still loses — and no mechanism decides that. It’s a question about whose intent wins, and it has to be answered by whoever owns the data.
So I haven’t solved this. Lost updates are among the best-documented problems in the field. What happens when one of the callers can read its own rejection and act on it is something I could find no published treatment of, which meant there was no external answer to test the model’s suggestions against. It’s a sequencing problem rather than a permissions problem, it’s uninteresting to almost everybody, and it is exactly the kind of thing that gets solved privately and stays that way.
The one that got outvoted
The second failure was different in kind. I was building OAuth authentication with Auth0 and Oracle ORDS — a fairly niche intersection with thin documentation. The database version I was running had only just been released, so the model could not have seen it: the relevant API had changed subtly, and the change post-dated anything it was trained on.
That was compounded by something structural. Unlike generated code, the model can’t inspect the database schema or Auth0’s internal state directly. It sees them through SQL and APIs, and both give a partial view, so it had no way to check its assumption against the system it was configuring.
Here’s the part that matters. Faced with a gap it couldn’t have filled, the model didn’t tell me it was out of date. It confidently produced the approach that used to be right, repeatedly, because there is an enormous amount of that approach in public and almost nothing yet about what replaced it. Absence in the corpus doesn’t produce silence. It produces the next most frequent answer, delivered with the same confidence as a correct one.
And then it defends it. When the approach didn’t work, the model did what a competent engineer does when their code fails: it assumed the mistake was its own and elaborated. Each attempt was a more intricate workaround built on the last — all locally reasonable, all of them working around a parameter it didn’t know existed. I backed every one of them out before applying the real fix. That’s the part that costs you, and it’s structural rather than careless: a model has no way to tell the difference between a problem it’s solving badly and a problem that cannot be solved the way it’s approaching it.
What ended it wasn’t a better prompt. It was a document — Kris Rice’s Oracle SKILLs — which I found and handed over, after many hours of experimentation and frustration. The correct approach was publicly available the whole time. It just wasn’t what the model produced until I put it in front of it.
There’s a name for the general shape. The TruthfulQA benchmark calls them imitative falsehoods: cases where a model reproduces a common misconception because the misconception is better represented in its training data than the correction. In the original experiments, the larger models tested were generally less truthful on that benchmark than smaller ones — a result the authors interpret as bigger models imitating the training distribution more completely, falsehoods included.
These are the two I can show you in detail. I’ve hit the same shape in performance work and in integration work, and I’d guess most people reading this have their own.
What the two have in common
Two failures, two mechanisms, and they’re worth keeping apart because they ask different things of you.
In the first, the answer is absent. Nobody wrote it down, and the reason is structural rather than accidental. Many enterprises don’t publish the postmortems, contracts and workarounds behind their most expensive failures — a hard-won fix is a differentiator, and it gets guarded. What reaches the public corpus is documentation, open source, standards, and millions of small examples: excellent for the typical case, silent on the expensive one.
In the second, the answer is outvoted. It exists in public — I found it — but it wasn’t in what the model had, and the gap didn’t announce itself. A thin or recent correct answer is statistically unlikely to be offered against a large volume of material that was correct when it was written and isn’t any more. Nothing in that older material was ever wrong. There’s just far more of it.
Retrieval is the obvious answer to the second case, and it’s precisely what I did by hand with the SKILLs document. That works — but only if you already know the model is wrong. The failure mode isn’t that it refuses; it’s that it answers well, in the right register, with the wrong thing.
Post-training seems to have taken the edge off the famous cases. Ask a model today whether we only use ten per cent of our brains and it will tell you no. But that attention goes to misconceptions someone judged worth correcting, and nobody is building a preference dataset for an ORDS API that shipped last quarter. The mechanism survives wherever the correction is too obscure to be worth anyone’s effort — which is a reasonable working definition of enterprise software.
So the useful question to ask before you hand something over isn’t whether it’s hard. Plenty of hard things are well documented and get built beautifully. The question is whether it’s written down, and whether the right version of it is winning.
Where it isn’t, nothing in the tool is going to tell you. It will keep going, confidently, in the wrong direction, and every hour it runs unwatched is an hour of unpicking afterwards. That’s the case against handing over a whole build and walking away — not that the model can’t be trusted, but that it has no way to recognise the boundary it has just walked into.