i build retrieval systems for companies who want their own documents answerable. ingest the knowledge base, embed it, retrieve against it, answer with citations, put a bot in front of it.
this one had 213 passing tests. the whole stack came up from a single terraform apply. every question in the golden evaluation set passed.
and in production it was quietly wrong in two different ways, neither of which raised a single error.
why rag failures are worse than crashes
when a service crashes you get an alert, a stack trace, a pager, a fix.
when a retrieval system fails it says “i don’t have that information” in a calm, well formatted, confident tone. or it gives you a vague answer that is technically about the right topic.
nobody files a bug for that. they conclude the documentation is incomplete, or that ai is overrated, and they stop using it. by the time it reaches you as a complaint you have already lost the users.
bug one: my parser ate the answer
the pipeline was fine. retrieval was pulling the right chunks. the model was generating a correct answer with correct citations.
then my citation parsing code ran over the output, hit a shape it did not expect, and replaced a correct answer with the fallback string. “i don’t have that information.”
so the system had the answer, generated the answer, and then threw it away at the last step.
this is the specific horror of the fallback string. it is designed to be a graceful failure, which means it is indistinguishable from a real miss. the logs looked normal. the evaluation set did not trigger the shape that broke the parser.
the fix took an hour. finding it took much longer, and it only got found because someone said “it definitely knows this, i wrote that page.”
bug two: the reranker was not running
the retrieval pipeline is hybrid. vector search plus bm25, deduped, then reranked, then sent to the model. reranking is what turns “these ten chunks are roughly relevant” into “these three are actually the answer”.
in production the reranker was not running at all. not because of a bug in the code but because of a permissions gap between the service and the reranking model. the call failed, the pipeline caught it, and it carried on with the unreranked results.
degrading gracefully was the correct engineering decision and it was also the reason nobody noticed for as long as we did not notice. answers were not wrong. they were just worse than they should have been, and worse is invisible without a baseline.
both of these now have monitoring. not on error rates, because neither of them produced errors. on the behaviour itself: is the reranker actually being called, and how often is the fallback string being emitted.
the third one was not even mine
there was a separate symptom. the bot was silently dropping some questions. not answering badly, not answering at all.
it was not the bot. it was the web application firewall.
an aws managed rule flags a request containing http:// plus a raw ip address as a remote file inclusion attempt, and returns a 403 before anything reaches the service. plain domains are fine. bare ip addresses are fine. the combination is what trips it.
people were pasting device urls into their questions. the firewall was reading a support question as an attack.
the fix was scoping that rule off the two free text question endpoints. the lesson is broader than the fix: when you put a text box in front of users and a firewall in front of the text box, every rule written for machine traffic is now a rule about human sentences.
what i changed in how i work
monitor behaviour, not errors. every optional stage of a pipeline needs a metric that says it ran. a stage that silently skips itself will silently skip itself for months.
the fallback string is a signal. track how often it fires. a sudden change in that rate is a bug, not a documentation gap.
your evaluation set is the questions you thought of. all three of these problems came from real users doing something my golden set did not contain. now real production questions feed back into the set.
check the layers you did not write. the firewall, the proxy, the load balancer. i spent hours inside my own code for a problem that lived three layers above it.
none of this makes the system smarter. it just makes it stop looking stupid in ways it did not have to.