Five RAG Architectures in Real Azure Code

Posted on September 13, 2026 by steefjan1970

Over the past few months I kept running into the similar looking infographics, in one form or another: five or six boxes, each a named RAG architecture, arrows showing how a query flows through it. Hybrid RAG. GraphRAG. Agentic RAG. Corrective RAG. Multimodal RAG. They’re useful as vocabulary. They are not implementation guides. None of them show you the part that actually takes the time.

So I built all five RAG architectures, on Azure, against one shared corpus and one shared set of test questions, and measured what came out. This post is the result: what each diagram leaves out, what the equivalent Azure code actually looks like, where the real deployment pitfalls were, and a comparison table built from real runs, not from argument.

The corpus is a fictional Dutch health insurer, Zorgverzekeraar Meridiaan, the same one I’ve used in a couple of other posts in this series. Nine documents: dental and physiotherapy policies, a provider network, an authorization process, a member complaint and the quarterly report that restates it, a stale FAQ sitting next to the current policy, a reimbursement table, and a scanned claim form. Fifteen questions, tagged by which pattern they were designed to stress. All five patterns answer all fifteen questions, so the comparison is apples to apples.

The repo is at github.com/steefjan1/five-rag-patterns if you want to run it yourself.

What the diagram shows vs. what the Azure code does

Hybrid RAG

The diagram draws dense and sparse retrieval as two separate paths that merge into a box labeled Reciprocal Rank Fusion. That box is mostly a non-event on Azure. Azure AI Search’s hybrid query type takes a vector query and a text query together and fuses them server-side. There is no RRF code to write.

What actually takes engineering effort is the index schema: chunk granularity (I chunk by document section, not by a fixed token window, so a retrieval unit is a coherent answer, not an arbitrary slice), which fields are filterable versus searchable versus vector, and whether semantic ranking is worth its cost on top of the fusion you already get for free.

Measured: recall 1.00 across all fifteen questions, the best of any pattern on pure coverage. Precision sits at 0.38, diluted by a fixed top-5 retrieval regardless of how many documents a question actually needs. Cheapest sane baseline in the set: $0.0026 and 2.00 seconds per query.

GraphRAG

The diagram draws one static graph: entities, edges, a subgraph retrieval step, a box for community summaries. What it doesn’t draw is that the graph has a maintenance cost. Community detection (Louvain, via networkx, which runs fine at this corpus’s scale without a dedicated graph database) and community summarization are real compute and real Azure OpenAI spend, paid once at setup and again every time the graph changes enough to shift community boundaries. Nothing about that shows up in the box-and-arrow version.

Entity linking here is a cheap substring match against entity names, not an embedding call, which is part of why this pattern is the cheapest per query in the whole set. Retrieval is a graph walk: two hops, both directions, so a question like “which hospital did this referral come from, and which GP group refers into that hospital” resolves correctly even though no single document states the answer. It’s two separate edges, walked in sequence.

Measured: recall 1.00 on its own three relational questions, the two-hop case included, and 0.21 on the other twelve. No other pattern swings that hard between its own territory and everything else. $0.0013 per query, the cheapest pattern here, in the narrowest lane.

Agentic RAG

The diagram shows a planner routing to tools and a reasoner that loops “until confident.” There is no upper bound drawn anywhere on that loop. Left alone, that is a cost leak, not a reliability feature, so the actual implementation caps it at five iterations and reports hitting the cap as its own outcome rather than quietly forcing an answer and calling it clean.

The other thing worth knowing if you’re building this on Azure: the AI Foundry Agent Service SDK bypasses API Management for its own LLM calls. I found this the hard way on an earlier project in this series. If your governance model depends on APIM, that means routing tool-calling agents through the standard OpenAI SDK pointed at the gateway, not through the framework’s own agent runtime, or every rate limit and kill switch you built stops applying the moment the agent framework makes the call instead of your code.

Two of this pattern’s four tools aren’t retrieval at all. The dental waiting-period and annual-maximum arithmetic is transcribed from the policy documents as plain code, not left for a language model to compute from prose. Insurance eligibility math is exactly the kind of thing an LLM gets subtly wrong under pressure, and exactly the kind of thing code gets right every time.

Measured: precision 1.00, recall 1.00 on its own two questions, and it’s the only pattern that doesn’t collapse elsewhere: recall 0.85 on the other thirteen, because it always has a general search tool as a fallback when nothing more specific fits. That’s the real finding here. It’s not that Agentic RAG is “better,” it’s that it hedges.

Corrective RAG

The diagram shows retrieve, grade, then three branches: answer, rewrite the query and loop back, or fall back to a web search. The rewrite loop has an arrow pointing backward and no stated exit condition. A closed corpus also has no web to fall back to, so “incorrect” here means declining to answer rather than guessing.

The corpus has a document built specifically to test the grading step: an archived FAQ with a plausible, wrong number sitting right next to the current policy with the right one. A pattern with no grading step retrieves both and may cite either. This one grades the retrieval, asks the model to identify which passage is authoritative using published dates and explicit supersession language, and only feeds the authoritative passages to the final answer. What got fetched and what got used are tracked separately on purpose, so a working grader shows zero distractor citations even though the distractor was retrieved.

Measured: precision 1.00, recall 1.00 on the three distractor questions, confirmed live, not just in the design. The more interesting number is that its other twelve questions score better (precision 0.79) than its own target slice (0.67). The grading discipline isn’t just catching the one distractor it was built to catch, it generalizes. That comes at a real cost: 3.97 seconds average latency, roughly double every other pattern, and the highest cost per query in the set, because a full run can mean three model calls instead of one.

Multimodal RAG

The diagram’s box says “shared multimodal embedding model (e.g. CLIP or ColPali),” which means self-hosting an embedding model. That’s a heavier operational commitment than anything else in this comparison needs, and it’s avoidable. This uses caption-then-embed instead: Document Intelligence extracts the actual structure of the reimbursement table (tables are exactly where a vision model hallucinates a plausible-looking row that isn’t in the source, so that step doesn’t get skipped), the vision-capable chat deployment captions the scanned claim form directly, and both captions get embedded with the same text-embedding-3-large deployment every other pattern uses. Same index Hybrid RAG built, two more documents in it, no new index and no new field.

One implementation note that cost real iteration: a first version of the captioning prompt asked for verbatim transcription, which correctly produced the form’s Dutch date format and Dutch status text. A validation step checking the caption against a hand-written ground truth flagged that as a mismatch, because the ground truth expected ISO dates and English. That’s not a captioning bug, it’s a prompt that needed to ask for normalization, not transcription. Worth deciding on purpose, since a shared index with mixed date formats and mixed languages retrieves worse than a normalized one.

Measured: recall 1.00 across the board and groundedness 1.00 on its own two questions, with no degradation on the other thirteen. That composability is the finding: this pattern is Hybrid RAG’s exact retrieve-and-answer loop plus two documents, and the numbers confirm that composition was free.

The comparison table

Same corpus, same fifteen questions, one pass, all five RAG architectures.

Pattern	Avg latency	Cost/query	Overall precision	Overall recall	Own-target recall
Hybrid	2.00s	$0.0026	0.38	1.00	1.00 (n=5)
GraphRAG	2.32s	$0.0013	0.20	0.37	1.00 (n=3)
Agentic	2.09s	$0.0045	0.45	0.87	1.00 (n=2)
Corrective	3.97s	$0.0051	0.77	0.97	1.00 (n=3)
Multimodal	2.18s	$0.0027	0.38	1.00	1.00 (n=2)

“Own-target” means the small subset of the fifteen questions each pattern was actually designed to answer (GraphRAG’s two-hop provider questions, Corrective RAG’s stale-document case, and so on). Every pattern hits recall 1.00 in its own lane. What separates them is what happens outside it: GraphRAG falls to 0.21 recall on the other twelve questions, Agentic RAG only falls to 0.85, and Hybrid, Corrective, and Multimodal don’t fall at all, because their retrieval isn’t scoped to a narrow entity set in the first place.

Two honest caveats on this table. Precision across every pattern is capped low by a fixed top-5 retrieval regardless of how many documents a question actually needs, so precision here measures retrieval breadth more than answer quality, read recall and the own-target column as the more meaningful columns. And the cost figures come from a placeholder price table, not a live Azure billing export, useful for comparing patterns against each other, not for a procurement conversation.

Deployment pitfalls

Every one of these was a real failure against a live Azure subscription, not a hypothetical.

Pinned model versions rot. A deployment written against gpt-4o-mini version 2024-07-18 failed eight months later with ServiceModelDeprecated. The fix wasn’t a newer pin, it was to stop pinning: leave the deployment’s model version empty and let Azure resolve the current default, and check az cognitiveservices model list -l <region> -o table before assuming a model name is still offered at all.

The account kind changed. Azure OpenAI is now provisioned through Foundry as kind: 'AIServices', not the older kind: 'OpenAI'. Same deployment mechanism underneath, different account kind and a newer API version. A template written against the old kind fails Cognitive Services preflight validation, not at compile time.

A malformed policy XML fails at ARM validation, not at Bicep build time. An APIM policy embedded as a Bicep string had a raw double-quoted path literal sitting inside an already double-quoted XML attribute. bicep build compiled it clean, because Bicep has no way to know a string is meant to be well-formed XML. The actual break only showed up against the live ARM validation API, after Azure AI Search, Cosmos DB, and the Foundry account had already finished provisioning. A small script that compiles the template and separately parses every embedded policy string as XML catches this before the next azd up, not during one.

RBAC role assignments alone don’t turn on Azure AD authentication. Azure AI Search kept returning a flat 403 on every data-plane call despite two correctly scoped role assignments, because the service still only accepted API-key authentication. Nothing had told it to accept AAD tokens at all. The fix is a separate property, disableLocalAuth: true, on the search service itself. If a resource with roles that look correct still refuses an authenticated caller, check the resource’s own auth settings before re-checking the role assignment.

Where none of this is the answer

None of these five patterns is the right first move for a small, stable knowledge base. Plain vector search, no fusion, no graph, no grading, no agent loop, is the correct answer until you can name the specific failure mode you’re buying insurance against. Every pattern here is a bet against one kind of failure, and every bet has a cost attached whether or not you ever collect on it.

Don’t build all five for one real system either. Pick based on the failure mode your domain actually has. GraphRAG only pays for itself if your questions are genuinely relational, multi-hop, the kind no single document answers. If they’re not, you’re paying setup cost and getting a narrower Hybrid RAG. Agentic RAG’s flexibility costs a planning call before any retrieval happens at all, worth it if your questions genuinely vary in shape, wasted overhead if they don’t. Corrective RAG’s discipline costs roughly double the latency of everything else in this comparison. That’s a fine trade when a wrong answer is expensive and a two-second wait isn’t. It’s a bad trade for a chat widget where speed is the product.

What this actually proves

The infographic’s taxonomy is real. These five RAG architectures are genuinely different, with genuinely different failure modes, and that part of the diagram holds up. What doesn’t hold up is the implication that the hard part is choosing between them. The hard part, in every case, was the piece the diagram didn’t draw: RRF turned out to be free because Azure AI Search already does it, but community detection is not free and has to be redone as the graph changes. An agent loop needs a hard cap or it’s an open-ended bill. A query rewrite loop needs the same cap for the same reason. A self-hosted multimodal embedding model turned out to be avoidable entirely, caption-then-embed onto infrastructure you already have gets you most of the way there.

The single most useful number in this whole exercise might be the smallest one: Corrective RAG’s grading step scored better on questions it wasn’t built for than on the one it was. That’s a pattern worth paying attention to. The things that make a RAG system more disciplined in one specific place often make it more disciplined everywhere, not just in the place you were testing for.

If you want to see the actual failure modes up close rather than the aggregate table, the earlier posts in this series go deeper on two of them: what naive RAG diagrams leave out covers the hybrid retrieval and groundedness gaps in more detail, and choosing between RAG, GraphRAG, and Agentic RAG when auditability is the constraint makes the conceptual case this post backs with numbers.

The full repo, including the corpus, the eval harness, and every pattern’s implementation, is at github.com/steefjan1/five-rag-patterns.

The Four Things Naive RAG Diagrams Leave Out

Posted on August 12, 2026 by steefjan1970

You might have seen the diagrams like four boxes, left to right, with indexing, retrieval, augmentation, and generation. Parse the PDF, chunk the text, embed the chunks, store the vectors. Then embed the question, search, stuff the results into a prompt, and generate. I have seen it circulating every few weeks with a fresh coat of branding and a caption promising an end to hallucination.

The diagram is not wrong. It is a decent first explanation of naive RAG. The problem, however, starts when someone treats it as a design. TThe version I saw recently ended its augmentation box with three words: zero hallucination guaranteed.

That claim is where I want to start, because it is the tell. Retrieval-augmented generation reduces fabrication. It does not eliminate it. Anyone promising zero has not yet run an evaluation against their own system.

So here are the four things the four-box picture leaves out, in the order they will hurt you. I have put runnable samples for each one in a companion repository.

Gap 1: Retrieval is not the same thing as vector search

Naive RAG diagrams draw a single arrow from question to embedding to vector database. That works beautifully in demos, because demo questions are written in the same register as the source documents.

Production questions are not. They contain product codes, policy numbers, abbreviations, proper nouns, and negations. Embeddings capture meaning, and a product code has no meaning to capture.

I built a small corpus to measure this rather than assert it: eight synthetic Dutch policy documents, two consecutive years of the same policy, a collective variant with a structurally identical pricing table under different codes, and a separate reglement for medical aids. Thirty-three chunks. Then eleven questions with a known correct chunk for each.

What vector-only missed

Vector-only retrieval got seven of the eleven right. The failures were not random:

Which discount applies to code BAS-VR-400? returned the document’s changes section, which names the code but never prices it. Right document, wrong section, and the retrieved chunk looks relevant enough to answer from.
Does medical acceptance apply to package AANV-CO-03 in 2026? returned the 2025 document, which says the opposite: right topic, wrong year, inverted answer.
How many physiotherapy treatments are in the Extra package in 2025? returned a chunk from the basic policy entirely.

That second one is the one that should worry you. It is not a near miss. The prose in the two years is nearly identical, the answer is reversed, and nothing downstream can tell. An assessor reading a fluent, cited, confidently wrong answer about acceptance criteria has no signal that anything went sideways.

Keyword search, meanwhile, nails the code lookups. Okapi BM25 has been solving this problem since before any of us had an opinion about transformers.

Worth being precise about what this does and does not prove. My first version of this corpus had three documents and eight chunks, and vector-only scored four out of five, because with eight chunks there is nothing to confuse. The gap only appears once the corpus contains things that genuinely resemble each other. If your own evaluation shows dense retrieval doing fine, check whether your test set is hard before concluding your pipeline is.

Fusion widens the pool, reranking picks the answer

Therefore, the answer is not to pick a side. Azure AI Search will run both and fuse the result lists with reciprocal rank fusion. Then a semantic reranker reorders the fused list using a cross-encoder that actually reads the query against each candidate.

Here is where my expectations were wrong, and where the measurement earned its keep. Across the eleven questions:

Strategy	Top-1 correct	MRR@5
Vector-only	7 / 11	0.77
Hybrid (BM25 + vector, RRF)	6 / 11	0.72
Hybrid + semantic reranker	11 / 11	1.00

Adding keyword search made it worse. Hybrid lost a case that vector-only got right, and fixed none.

Why fusion alone went backwards

The reason is visible in the failures. BM25 matches the literal string BAS-VR-350, and that code appears in the document’s changes section, which names codes without pricing them. Lexical matching therefore promoted chunks that contain the code and cannot answer the question. Reciprocal rank fusion then faithfully merged two ranked lists, because RRF has no notion of whether a chunk answers anything. It fuses positions, not relevance.

The cross-encoder is what fixed it. It reads the question against each candidate and understands that a question about a discount needs the row with a price in it, not the sentence announcing that the code exists. That took the same candidate set from six correct to eleven.

So the lesson is sharper than “use hybrid search”. Hybrid retrieval widens the candidate pool; reranking is what converts a wider pool into better answers. Ship the first without the second, and you may go backward quietly, because nothing in the pipeline reports that it happened.

One clean question per turn is an assumption

Query handling is the other half of this gap. The diagram assumes one clean question per turn. Real questions arrive compound: we switched to the Compleet package in March, does my son’s dental work fall under that or under the basic policy, and does the deductible apply? That is three questions. Embed the whole sentence, and you retrieve the average of three intents, which is nothing in particular.

Agentic retrieval in Azure AI Search handles that by decomposing the query into subqueries, running them in parallel, reranking each, and merging. Extractive retrieval went generally available in API version 2026-04-01. Query planning and answer synthesis remain preview. Worth knowing which half you are depending on before you promise it to a steering committee.

Gap 2: Chunking is most of the work

“Chunk text for sharp recall.” One bullet. In practice, this single decision determines more of your answer quality than your choice of model.

Fixed-size splitting is what every quickstart does and what almost nothing should do. Run a 400-character window with 50 characters of overlap over a document containing a pricing table, and the splitter lands mid-row. This is the actual output from the sample, not an illustration:

                        | EUR 3,00          | BAS-VR-100  |
| EUR 200                 | EUR 6,50          | BAS-VR-200  |
| EUR 300                 | EUR 10,00         | BAS-VR-300  |
| EUR 400                 | EUR 14,00         | BAS-VR-400  |
| EUR 500                 | EUR 19,00         | BAS-VR-500  |

## 2. Fysiotherapie

Fysiotherapie wordt vanaf de 21e

Look at what survived. No header row, so nothing says which column is the deductible, which is the monthly discount, and which is the product code. The first row is cut mid-cell: its deductible tier is gone, leaving a discount attached to nothing. No document title, so nothing says this is the 2026 basic policy rather than the 2025 one or the collective variant, all three of which carry a table of exactly this shape with different numbers. And the chunk runs on into an unrelated section about physiotherapy, ending mid-sentence.

Retrieve that, and the model has to guess. It will guess. It will sound certain. And the citation attached to it will make the wrong answer more credible, not less.

What structure-aware chunking does differently

The sample runs three chunkers over the same document and counts how many chunks ended up holding table rows with no header. Fixed-size produces one out of three. Recursive paragraph splitting produces none, but leaves every chunk without a section heading. Structure-aware produces four chunks, none headerless, none context-free.

The difference is three rules, and none of them is clever: split on headings rather than character counts, never split a table, and prepend the document title and section heading to every chunk so an isolated chunk still says what it is.

That last rule is what makes the three near-identical pricing tables in this corpus distinguishable at all. Without it, retrieval has to tell them apart on the numbers alone.

Gap 3: Retrieval without authorization is a breach with a chat interface

This is the gap that should worry you most, and it is absent from every version of the diagram I have seen.

Put every document in one index. Wire up a chat interface. Now every user can reach every document, because semantic search does not know about your authorization model. The retrieval layer will happily surface an internal work instruction, an HR file, or a legal memo to whoever asks a question shaped roughly like its contents.

The filter is a query construct, not a prompt instruction

Two things follow. First, the filter is a server-side query construct, not a prompt instruction. Telling the model “only use documents the user may see” is not a control; it is a suggestion to a system that has already been handed the text. Second, the filter must derive from validated token claims, never from anything the user typed.

Two mechanisms, and the one that fails open

Azure AI Search gives you two mechanisms. The durable one is an explicit filterable collection of group identifiers on each document plus an OData filter built from the caller’s claims, which works today on the stable API and which you own end to end. The managed one ingests RBAC scopes, ACLs, or Purview sensitivity labels alongside the content and enforces them at query time when you pass the user’s token in the x-ms-query-source-authorization header.

The managed route has a sharp edge worth memorizing. If the knowledge source was created without ingestionPermissionOptions, the index holds no permission metadata, and results come back unfiltered regardless of the header. It fails open quietly, and the only way to fix it is to recreate the knowledge source. As of the current GA release, document-level permissions on indexed sources remain in preview.

Whichever you choose, write the leak test. The sample repository includes one: a query, an unauthorized caller, and an assertion that fails the build if the restricted document comes back. Twelve lines. Run it in CI.

Gap 4: “Zero hallucination” is a claim, and claims get measured

Grounding the prompt does not guarantee a grounded answer. Three failure modes survive the diagram intact.

The model can prefer what it already knows over what you retrieved. Ask about a monthly premium that appears nowhere in your corpus, and a model trained on the open internet has plausible Dutch premiums available. It will produce one.

The model can blend two chunks into a claim neither of them makes. This is the subtle one, because every individual fact traces back to a source.

And the model can answer confidently when retrieval returned nothing relevant at all, because nothing in the naive RAG pipeline tells it that “I do not know” is an available output.

Three rules that make it measurable

Consequently, the fix is threefold and unexciting: an explicit refusal string in the prompt so refusal is detectable rather than inferred, mandatory citation of a reference identifier after every claim so each statement is checkable, and an evaluation set that contains questions your corpus cannot answer.

That last point is the one most teams skip. Everyone builds a golden set of questions the documents answer well. Almost nobody includes the withdrawn product code, the topic that was never documented, or the answer that lives in a file the user may not see. Those are exactly the cases that generate the incident report.

Score retrieval and generation separately, too. A wrong answer with good retrieval is a generation problem. A wrong answer with bad retrieval is a retrieval problem. Without both numbers, you will spend a week tuning the wrong half of the system.

What the numbers actually said

Then the result, which surprised me: fourteen out of fourteen on groundedness and valid citations, and five out of five refusals on the unanswerable questions. The model corrected false premises rather than accepting them; asked whether medical acceptance applied in 2026, it answered yes and cited; asked about a product code withdrawn before the corpus begins, it refused outright rather than interpolating a plausible price from the neighboring rows.

I want to be careful about what that number means, because it is easy to oversell in the other direction. Retrieval hit fourteen out of fourteen in the same run, so generation was working from good material throughout. This is not evidence that hallucination is solved. It is evidence that the three unexciting mechanisms above are sufficient when retrieval is doing its job, which is the argument for building all four gaps rather than any one of them.

One caveat I would want stated if someone showed me this number: the judge was the same model as the generator. A model scoring its own output shares its own blind spots, and the groundedness figure is inflated to an unknown degree by that. Use a different model for judging if the number needs to carry weight.

Where this is the wrong answer

If your corpus is fifty pages of prose, with no product codes, no tables, one audience, and low stakes, then the four-box diagram is enough. Hybrid retrieval, structure-aware chunking, security trimming, and a groundedness harness are all overhead you do not need yet. Build the simple thing, ship it, and see what breaks.

The four gaps become urgent at specific, recognizable moments: the first exact-match question that returns the wrong table row, the first document that should not be visible to everyone, and the first stakeholder who asks how you know the answers are right. If none of those have happened, you are fine.

The part the diagram gets right

Context beats prompt engineering. That much is true, and it is the reason the picture keeps circulating. But “give the model the right information at the right time” restates the problem. It is not a solution. The right information depends on hybrid retrieval and reranking. The right time depends on query decomposition. Whether the user was entitled to that information depends on trimming. And whether the model actually used it depends on evaluation.

Do all four and the numbers hold up — mine did. Skip any one of them, and you will not find out which one you skipped until someone asks a question about last year’s policy and gets this year’s answer.

Four boxes, four gaps. The gaps are where the engineering lives.

The samples are at steefjan1/naive-rag-gap: hybrid retrieval and reranking, chunking, security trimming, and groundedness evaluation, provisioned with a single azd up. Four of the five run end to end against a live service; the agentic retrieval sample needs a knowledge base that none of the scripts create, so treat that one as a sketch rather than a worked example.

They target Azure AI Search API versions current as of August 2026. Agentic retrieval and document-level permissions are both moving quickly, so check the docs before assuming a preview flag is still a preview flag.

Cloud Perspectives

Steef-Jan Wiggers

Tag Archives: Azure AI Search