Andrews DeanThe fix passed every test I wrote for it. It still would have made the product worse than the bug it replaced, and I only found that out because I checked one number I didn't strictly have to check.
A rate limit, and an obvious fix
An evaluation suite I'd built for an AI agent failed four cases on its first real run, out of thirty four. Every failure traced back to the same cause: the embedding provider's free tier allows three requests a minute, and a full evaluation run hammers it far harder than that. Not a bug in the agent. A quota.
The fix suggested itself immediately. If the primary embedding provider fails, quietly retry the same call through a second one instead. Nobody sees an error, the eval run finishes clean, and the whole thing feels like exactly the kind of resilience a production system is supposed to have.
I built it. It passed its unit tests without complaint.
The number that shouldn't have been possible
Before shipping it, I ran it against the real document corpus instead of trusting the tests, mostly out of habit. That one habit is the entire reason this essay exists.
A question that should score somewhere around 0.3 to 0.5 similarity against its correct answer, the normal range for a real match, came back at 0.03. Not a weak match. Noise. The kind of number you'd get comparing two things that have nothing to do with each other.
The cause took a while to accept, because it isn't intuitive. Two different embedding models can output vectors of the exact same length, filled with numbers that look equally precise and equally valid, and still not share a coordinate space. It's two maps of the same city, drawn at the same scale, each with its own starting point for the grid. A coordinate from one map isn't approximately right on the other. It's meaningless on the other, and it will still look like a real coordinate right up until you try to use it.
The corpus was embedded entirely by the primary provider. The fallback quietly answered one question using the second provider instead, mid-run, and produced a number that looked exactly as legitimate as a real similarity score.
Worse than the bug it replaced
Here's the part that actually mattered. The system didn't crash, and it didn't return an error. It confidently reported that the question wasn't covered by the policy, for a question the policy quite clearly did cover. A hallucination's quieter cousin: not inventing an answer, just inventing an absence.
I'd built the fallback to make the system more reliable. It made the system reliably wrong, and dressed the wrongness up as caution, which is a harder failure to catch than an honest crash, because nothing about it asks to be investigated.
Let me put the distinction plainly, because it's the whole lesson: a test proving the code path runs is not the same claim as a test proving the result means something. The first kind of test is exactly what I'd already passed. The second kind is the only one that would have caught this, and it only works if you run it against real data instead of a mock that can't possibly know two embedding spaces disagree.
What I did about it
I reverted the fallback completely. Not disabled, not flagged, gone, and I wrote the actual numbers, 0.03 against an expected 0.3 to 0.5, straight into the embedding code's own documentation, specifically so that nobody rediscovers this the hard way and quietly re-adds a version of the same fix six months from now.
The fix that would actually work, embedding every document with both providers ahead of time so a fallback never has to cross a boundary it can't see, is real work, and I wrote it down as exactly that: scoped, not built, on the list. Shipping a half-solution under time pressure so it looks finished is its own kind of failure, and it's one I'd rather admit to than hide.
So here's the question worth sitting with, whatever you're shipping this week. Which of your tests prove the code ran, and which ones prove the answer was actually true, and would you know the difference if the wrong one started passing?