You might have seen the diagrams like four boxes, left to right, with indexing, retrieval, augmentation, and generation. Parse the PDF, chunk the text, embed the chunks, store the vectors. Then embed the question, search, stuff the results into a prompt, and generate. I have seen it circulating every few weeks with a fresh coat of branding and a caption promising an end to hallucination.

The diagram is not wrong. It is a decent first explanation of naive RAG. The problem, however, starts when someone treats it as a design. TThe version I saw recently ended its augmentation box with three words: zero hallucination guaranteed.

That claim is where I want to start, because it is the tell. Retrieval-augmented generation reduces fabrication. It does not eliminate it. Anyone promising zero has not yet run an evaluation against their own system.

So here are the four things the four-box picture leaves out, in the order they will hurt you. I have put runnable samples for each one in a companion repository.

Gap 1: Retrieval is not the same thing as vector search

Naive RAG diagrams draw a single arrow from question to embedding to vector database. That works beautifully in demos, because demo questions are written in the same register as the source documents.

Production questions are not. They contain product codes, policy numbers, abbreviations, proper nouns, and negations. Embeddings capture meaning, and a product code has no meaning to capture.

I built a small corpus to measure this rather than assert it: eight synthetic Dutch policy documents, two consecutive years of the same policy, a collective variant with a structurally identical pricing table under different codes, and a separate reglement for medical aids. Thirty-three chunks. Then eleven questions with a known correct chunk for each.

What vector-only missed

Vector-only retrieval got seven of the eleven right. The failures were not random:

• Which discount applies to code BAS-VR-400? returned the document’s changes section, which names the code but never prices it. Right document, wrong section, and the retrieved chunk looks relevant enough to answer from.
• Does medical acceptance apply to package AANV-CO-03 in 2026? returned the 2025 document, which says the opposite: right topic, wrong year, inverted answer.
• How many physiotherapy treatments are in the Extra package in 2025? returned a chunk from the basic policy entirely.

That second one is the one that should worry you. It is not a near miss. The prose in the two years is nearly identical, the answer is reversed, and nothing downstream can tell. An assessor reading a fluent, cited, confidently wrong answer about acceptance criteria has no signal that anything went sideways.

Keyword search, meanwhile, nails the code lookups. Okapi BM25 has been solving this problem since before any of us had an opinion about transformers.

Worth being precise about what this does and does not prove. My first version of this corpus had three documents and eight chunks, and vector-only scored four out of five, because with eight chunks there is nothing to confuse. The gap only appears once the corpus contains things that genuinely resemble each other. If your own evaluation shows dense retrieval doing fine, check whether your test set is hard before concluding your pipeline is.

Fusion widens the pool, reranking picks the answer

Therefore, the answer is not to pick a side. Azure AI Search will run both and fuse the result lists with reciprocal rank fusion. Then a semantic reranker reorders the fused list using a cross-encoder that actually reads the query against each candidate.

Here is where my expectations were wrong, and where the measurement earned its keep. Across the eleven questions:

Strategy	Top-1 correct	MRR@5
Vector-only	7 / 11	0.77
Hybrid (BM25 + vector, RRF)	6 / 11	0.72
Hybrid + semantic reranker	11 / 11	1.00

Adding keyword search made it worse. Hybrid lost a case that vector-only got right, and fixed none.

Why fusion alone went backwards

The reason is visible in the failures. BM25 matches the literal string BAS-VR-350, and that code appears in the document’s changes section, which names codes without pricing them. Lexical matching therefore promoted chunks that contain the code and cannot answer the question. Reciprocal rank fusion then faithfully merged two ranked lists, because RRF has no notion of whether a chunk answers anything. It fuses positions, not relevance.

The cross-encoder is what fixed it. It reads the question against each candidate and understands that a question about a discount needs the row with a price in it, not the sentence announcing that the code exists. That took the same candidate set from six correct to eleven.

So the lesson is sharper than “use hybrid search”. Hybrid retrieval widens the candidate pool; reranking is what converts a wider pool into better answers. Ship the first without the second, and you may go backward quietly, because nothing in the pipeline reports that it happened.

One clean question per turn is an assumption

Query handling is the other half of this gap. The diagram assumes one clean question per turn. Real questions arrive compound: we switched to the Compleet package in March, does my son’s dental work fall under that or under the basic policy, and does the deductible apply? That is three questions. Embed the whole sentence, and you retrieve the average of three intents, which is nothing in particular.

Agentic retrieval in Azure AI Search handles that by decomposing the query into subqueries, running them in parallel, reranking each, and merging. Extractive retrieval went generally available in API version 2026-04-01. Query planning and answer synthesis remain preview. Worth knowing which half you are depending on before you promise it to a steering committee.

Gap 2: Chunking is most of the work

“Chunk text for sharp recall.” One bullet. In practice, this single decision determines more of your answer quality than your choice of model.

Fixed-size splitting is what every quickstart does and what almost nothing should do. Run a 400-character window with 50 characters of overlap over a document containing a pricing table, and the splitter lands mid-row. This is the actual output from the sample, not an illustration:

                        | EUR 3,00          | BAS-VR-100  |
| EUR 200                 | EUR 6,50          | BAS-VR-200  |
| EUR 300                 | EUR 10,00         | BAS-VR-300  |
| EUR 400                 | EUR 14,00         | BAS-VR-400  |
| EUR 500                 | EUR 19,00         | BAS-VR-500  |

## 2. Fysiotherapie

Fysiotherapie wordt vanaf de 21e

Look at what survived. No header row, so nothing says which column is the deductible, which is the monthly discount, and which is the product code. The first row is cut mid-cell: its deductible tier is gone, leaving a discount attached to nothing. No document title, so nothing says this is the 2026 basic policy rather than the 2025 one or the collective variant, all three of which carry a table of exactly this shape with different numbers. And the chunk runs on into an unrelated section about physiotherapy, ending mid-sentence.

Retrieve that, and the model has to guess. It will guess. It will sound certain. And the citation attached to it will make the wrong answer more credible, not less.

What structure-aware chunking does differently

The sample runs three chunkers over the same document and counts how many chunks ended up holding table rows with no header. Fixed-size produces one out of three. Recursive paragraph splitting produces none, but leaves every chunk without a section heading. Structure-aware produces four chunks, none headerless, none context-free.

The difference is three rules, and none of them is clever: split on headings rather than character counts, never split a table, and prepend the document title and section heading to every chunk so an isolated chunk still says what it is.

That last rule is what makes the three near-identical pricing tables in this corpus distinguishable at all. Without it, retrieval has to tell them apart on the numbers alone.

Gap 3: Retrieval without authorization is a breach with a chat interface

This is the gap that should worry you most, and it is absent from every version of the diagram I have seen.

Put every document in one index. Wire up a chat interface. Now every user can reach every document, because semantic search does not know about your authorization model. The retrieval layer will happily surface an internal work instruction, an HR file, or a legal memo to whoever asks a question shaped roughly like its contents.

The filter is a query construct, not a prompt instruction

Two things follow. First, the filter is a server-side query construct, not a prompt instruction. Telling the model “only use documents the user may see” is not a control; it is a suggestion to a system that has already been handed the text. Second, the filter must derive from validated token claims, never from anything the user typed.

Two mechanisms, and the one that fails open

Azure AI Search gives you two mechanisms. The durable one is an explicit filterable collection of group identifiers on each document plus an OData filter built from the caller’s claims, which works today on the stable API and which you own end to end. The managed one ingests RBAC scopes, ACLs, or Purview sensitivity labels alongside the content and enforces them at query time when you pass the user’s token in the x-ms-query-source-authorization header.

The managed route has a sharp edge worth memorizing. If the knowledge source was created without ingestionPermissionOptions, the index holds no permission metadata, and results come back unfiltered regardless of the header. It fails open quietly, and the only way to fix it is to recreate the knowledge source. As of the current GA release, document-level permissions on indexed sources remain in preview.

Whichever you choose, write the leak test. The sample repository includes one: a query, an unauthorized caller, and an assertion that fails the build if the restricted document comes back. Twelve lines. Run it in CI.

Gap 4: “Zero hallucination” is a claim, and claims get measured

Grounding the prompt does not guarantee a grounded answer. Three failure modes survive the diagram intact.

The model can prefer what it already knows over what you retrieved. Ask about a monthly premium that appears nowhere in your corpus, and a model trained on the open internet has plausible Dutch premiums available. It will produce one.

The model can blend two chunks into a claim neither of them makes. This is the subtle one, because every individual fact traces back to a source.

And the model can answer confidently when retrieval returned nothing relevant at all, because nothing in the naive RAG pipeline tells it that “I do not know” is an available output.

Three rules that make it measurable

Consequently, the fix is threefold and unexciting: an explicit refusal string in the prompt so refusal is detectable rather than inferred, mandatory citation of a reference identifier after every claim so each statement is checkable, and an evaluation set that contains questions your corpus cannot answer.

That last point is the one most teams skip. Everyone builds a golden set of questions the documents answer well. Almost nobody includes the withdrawn product code, the topic that was never documented, or the answer that lives in a file the user may not see. Those are exactly the cases that generate the incident report.

Score retrieval and generation separately, too. A wrong answer with good retrieval is a generation problem. A wrong answer with bad retrieval is a retrieval problem. Without both numbers, you will spend a week tuning the wrong half of the system.

What the numbers actually said

Then the result, which surprised me: fourteen out of fourteen on groundedness and valid citations, and five out of five refusals on the unanswerable questions. The model corrected false premises rather than accepting them; asked whether medical acceptance applied in 2026, it answered yes and cited; asked about a product code withdrawn before the corpus begins, it refused outright rather than interpolating a plausible price from the neighboring rows.

I want to be careful about what that number means, because it is easy to oversell in the other direction. Retrieval hit fourteen out of fourteen in the same run, so generation was working from good material throughout. This is not evidence that hallucination is solved. It is evidence that the three unexciting mechanisms above are sufficient when retrieval is doing its job, which is the argument for building all four gaps rather than any one of them.

One caveat I would want stated if someone showed me this number: the judge was the same model as the generator. A model scoring its own output shares its own blind spots, and the groundedness figure is inflated to an unknown degree by that. Use a different model for judging if the number needs to carry weight.

Where this is the wrong answer

If your corpus is fifty pages of prose, with no product codes, no tables, one audience, and low stakes, then the four-box diagram is enough. Hybrid retrieval, structure-aware chunking, security trimming, and a groundedness harness are all overhead you do not need yet. Build the simple thing, ship it, and see what breaks.

The four gaps become urgent at specific, recognizable moments: the first exact-match question that returns the wrong table row, the first document that should not be visible to everyone, and the first stakeholder who asks how you know the answers are right. If none of those have happened, you are fine.

The part the diagram gets right

Context beats prompt engineering. That much is true, and it is the reason the picture keeps circulating. But “give the model the right information at the right time” restates the problem. It is not a solution. The right information depends on hybrid retrieval and reranking. The right time depends on query decomposition. Whether the user was entitled to that information depends on trimming. And whether the model actually used it depends on evaluation.

Do all four and the numbers hold up — mine did. Skip any one of them, and you will not find out which one you skipped until someone asks a question about last year’s policy and gets this year’s answer.

Four boxes, four gaps. The gaps are where the engineering lives.

The samples are at steefjan1/naive-rag-gap: hybrid retrieval and reranking, chunking, security trimming, and groundedness evaluation, provisioned with a single azd up. Four of the five run end to end against a live service; the agentic retrieval sample needs a knowledge base that none of the scripts create, so treat that one as a sketch rather than a worked example.

They target Azure AI Search API versions current as of August 2026. Agentic retrieval and document-level permissions are both moving quickly, so check the docs before assuming a preview flag is still a preview flag.