Your RAG isn't bad. Your chunking is.
Most retrieval failures get blamed on the model. In production they almost always trace back to how the documents were split — and that is a much cheaper thing to fix.
Every team I review has the same conversation in the same order. Retrieval quality is bad, so they try a bigger embedding model. Still bad, so they try a reranker. Still bad, so they start pricing a fine-tune.
Almost nobody goes back and looks at what the chunks actually contain.
The failure mode
A fixed-size splitter at 512 tokens with a 50-token overlap will happily cut a table in half, separate a heading from the paragraph it introduces, and split a numbered procedure across three chunks so that step 4 arrives without steps 1 to 3. The embedding of that fragment is a perfectly good embedding — of a fragment that means nothing on its own.
Then retrieval returns it, the model reads it, and the model does what models do with insufficient context: it fills the gap.
What to check before you touch the model
- Pull twenty retrieved chunks at random and read them as a human. If you cannot answer the question from the chunk, neither can the model.
- Check whether your splitter respects document structure. Headings, tables, list items and code blocks are semantic units. Cutting through them destroys meaning.
- Check what metadata survives. A chunk with no document title, no section path and no date is much harder to rank and impossible to cite properly.
- Measure retrieval separately from generation. If you only look at the final answer you cannot tell which half is failing.
The fix is usually boring
Structure-aware splitting, a parent-document strategy so the model sees the surrounding section, and metadata carried through into the index. It is a day of work on the ingestion pipeline rather than a month on the model.
That is the general pattern with production LLM systems: the expensive-looking problem is usually a cheap problem in the data layer wearing a costume.