A RAG pipeline that actually answers from the manual

Naïve RAG returns plausible answers that are not in the manual. This is the failure mode that kills production RAG, because plausible-but-wrong is worse than "I don't know". The customer gets dispatched a technician for the wrong part; the company loses a day of revenue and a unit of trust it cannot get back.

The Extrema customer-support chatbot started naïve. We chunked the manuals into 512-token blocks, embedded them with OpenAI ada-002, retrieved top-5 by cosine, stuffed them into a GPT-4 prompt with "answer based on these documents". The demo was magnificent. The first week in production it confidently told a customer that the LV-450 controller could be hot-swapped — it cannot — because a paragraph about a different controller used similar wording.

The fix took three boring layers and no new model. First, section-aware chunking. The H2 heading above a paragraph is half the signal. A chunk that says "Hot-swap procedure" inside a section titled "LV-380 maintenance" should not be retrievable as an answer to a question about the LV-450. We re-chunked to keep the full breadcrumb (Manual → Chapter → Section → Subsection) as a structural prefix on every chunk, and let the retriever see it.

Second, hybrid retrieval. Pure embedding retrieval is great for paraphrase, terrible for exact-term matches like part numbers. We added BM25 over the same chunks and did a reciprocal-rank fusion of the two result lists, then re-ranked the top 20 with a small cross-encoder. The re-ranker is the unsung hero of production RAG. It costs 20 ms per query and prevents the embedding from confidently returning the wrong section.

Third, an answer template that quotes the manual or refuses. The system prompt instructs the model to always cite the manual page number for any factual claim, and to respond with a structured "I cannot find this in the manuals" if no retrieved chunk supports the answer. The output a customer sees is dull: "According to Manual page 142, step 4: turn the breaker off, wait 60 seconds, then unscrew the cover." The dullness is the feature. Boring grounded answers, with citations, are what builds trust over weeks.

The production impact at Extrema: internal customer-service response time went down by more than 60%, mostly because the support engineer no longer has to flip through 1,400 pages to find the right procedure — the bot has already done it, with a page reference. Customers who self-serve get the same answer the engineer would have given them, faster. And critically, on questions where the bot does not know, it says so, and the case escalates to a human within seconds rather than after a wrong answer has already done damage.

The single biggest takeaway from the project is that the headline metric in RAG is not retrieval recall. It is "did we refuse when we should have refused". Optimise for that and the rest follows.