Building RAG Pipelines with Azure Functions: Event-Driven Data Retrieval at Scale

Posted on August 19, 2026 by steefjan1970

This is the fifth and final post in the series on AI and Azure Functions. The first post mapped all four AI-enabled patterns. Posts two through four went deep on MCP server hosting, the serverless agents runtime, and Durable Functions for directed agentic workflows. This post covers the fourth pattern: retrieval-augmented generation.

RAG is where Azure Functions earns its place through a capability that is easy to underestimate: the ability to handle multiple events from multiple data sources simultaneously. A RAG pipeline is only as good as its retrieval layer, and retrieval latency in production comes down to two things: how fast you can query your data sources, and how fast you can scale when demand spikes. Azure Functions on Flex Consumption handles both.

What RAG on Azure Functions actually looks like

The Microsoft Learn documentation is deliberately brief on this pattern: “RAG systems require fast data retrieval and processing. Functions can interact with multiple data sources simultaneously and provide the rapid scale required by RAG scenarios.”

That one-liner contains three architectural decisions worth unpacking.

Multiple data sources simultaneously. A retrieval layer that queries one source at a time is a bottleneck. A function app can fan out — trigger parallel calls to Azure AI Search, a Cosmos DB vector store, a blob-indexed knowledge base, and a case history API — collect all results, and pass the aggregated context to the language model in a single call. This is the fan-out/fan-in pattern from the Durable Functions post, applied to retrieval rather than to a directed workflow.

Event-driven. The most common RAG pattern in tutorials is synchronous: the user sends a query, retrieval runs, and the LLM responds. But production RAG often has an asynchronous dimension — new documents arrive, get processed, get indexed. Azure Functions handles both sides: HTTP-triggered retrieval for the synchronous path, and blob/Event Hubs/Cosmos DB change feed triggers for the ingestion pipeline that keeps the index current. The same function app can serve both.

Rapid event-driven scaling. The Azure OpenAI binding extension adds stateful chat session support — the function maintains conversation history across turns without you writing session management code. Combined with Flex Consumption’s scale-to-zero billing, that means a RAG-enabled chat endpoint costs nothing when idle and scales in seconds when traffic arrives.

The Azure OpenAI binding extension

The binding extension is the most concrete piece of infrastructure the docs call out for RAG. It provides three bindings relevant to RAG scenarios:

Text completion input binding — calls an Azure OpenAI deployment and returns the response. Used for one-shot prompt/response patterns.

Chat completion input binding — maintains a stateful chat session across function invocations. The session history is stored externally (Azure Table Storage by default) and automatically injected into each prompt. This is what the “Custom chat bot” sample in the docs demonstrates.

Embeddings input binding — generates vector embeddings from text. Used in the ingestion pipeline to embed documents before writing them to a vector store.

A simple RAG function using the chat completion binding looks like this in C#:

			
[Function(nameof(RagChat))]
public static async Task<IActionResult> RagChat(
    [HttpTrigger(AuthorizationLevel.Function, "post")] HttpRequestData req,
    [TextCompletion(
        "{query}",
        Model = "%OPENAI_DEPLOYMENT%",
        SystemPrompt = "You are a helpful assistant. Answer using only the context provided."
    )] TextCompletionResponse completion)
{
    return new OkObjectResult(completion.Content);
}

		

The binding handles the Azure OpenAI call, authentication via managed identity, and response parsing. You write the retrieval logic — what context to inject into {query} — and the binding handles the LLM call.

The production version adds a retrieval step before the binding executes. In practice, this means the HTTP trigger extracts the user query, a call to Azure AI Search (or your vector store of choice) retrieves relevant passages, those passages are injected into the prompt template, and then the binding submits the augmented prompt to the LLM.

APIM as the LLM gateway layer

The binding extension simplifies the function code, but it routes directly to Azure OpenAI — there is no governance layer between the function and the model by default. For production RAG pipelines, that matters: you need rate limiting per client, token quota management, cost attribution across tenants or use cases, and a single point for monitoring and logging LLM calls.

APIM fills that role. Place it between the function app and Azure OpenAI, configure an inbound policy to validate the request and apply per-subscription token quotas, and configure an outbound policy to log usage. The function app calls the APIM endpoint rather than the Azure OpenAI endpoint directly.

One practical constraint from real deployment experience: if you use the Azure AI Foundry Agent Service SDK alongside Azure Functions, the SDK routes LLM calls directly to Azure OpenAI and bypasses APIM entirely. The standard OpenAI SDK doesn’t behave this way—calls flow through APIM as expected. The Citadel Platform series covers this in detail, including how to structure APIM policies for the token quota patterns that matter in production.

For RAG pipelines specifically, two APIM policies are worth implementing:

Token quota by subscription key. Set a per-day or per-hour token budget for each API consumer. RAG queries are often significantly larger than simple completions because the retrieved context inflates the prompt — a query with 2,000 tokens of retrieved passages consumes far more quota than a direct LLM call. Factor this into your quota design.

Semantic caching. APIM’s semantic caching policy stores LLM responses and returns cached results for semantically similar queries. For a RAG pipeline serving a knowledge base that changes infrequently, this can dramatically reduce token consumption and latency.

The ingestion pipeline

The retrieval side of RAG gets most of the attention, but the ingestion pipeline- how documents get into the index is where Azure Functions adds less-discussed value.

When a new document arrives in blob storage, the function extracts text, generates a vector embedding, and writes the result to Azure AI Search. An Event Hubs trigger handles high-throughput ingestion where many documents arrive simultaneously. A Cosmos DB change feed trigger keeps a vector index in sync with a transactional database.

The key advantage over a dedicated ingestion service is that the same managed identity, Application Insights workspace, and azd deployment pipeline cover both retrieval and ingestion. Operational complexity stays low even as the pipeline grows.

One constraint to know before building: blob triggers on Flex Consumption require EventGrid as the source. Standard polling blob triggers are not supported on the Flex Consumption plan. This means you must configure an EventGrid system topic and event subscription after provisioning — a two-step post-provision operation that cannot be included in the same Bicep deployment as the function app.

A minimal ingestion function using the Azure OpenAI SDK for embeddings — the recommended approach until the embeddings binding extension reaches GA:

			
[Function(nameof(IngestDocument))]
public async Task Run(
    [EventGridTrigger] EventGridEvent eventGridEvent,
    CancellationToken cancellationToken)
{
    // Read blob from storage, generate embedding, write to Azure AI Search
    var blobContent = await ReadBlobAsync(eventGridEvent.Subject, cancellationToken);
    var embeddingClient = openAiClient.GetEmbeddingClient(embeddingsDeployment);
    var embedding = await embeddingClient.GenerateEmbeddingAsync(
        blobContent, cancellationToken: cancellationToken);
    var document = new SearchDocument { ContentVector = embedding.Value.ToFloats().ToArray() };
    await searchClient.IndexDocumentsAsync(
        IndexDocumentsBatch.Upload([document]), cancellationToken: cancellationToken);
}

		

The EventGrid trigger fires on MicrosoftStorage.BlobCreated events; the function reads the blob, generates the vector, and writes to the index. EventGrid delivery includes retry logic —failed events are retried with exponential backoff for up to 24 hours.

Connecting the four patterns

This is the final post in the series, so it is worth mapping how the four AI patterns relate to each other in a production architecture. They are not mutually exclusive.

A realistic enterprise RAG system might use all four:

Azure Functions RAG pipeline handles retrieval — HTTP-triggered queries fan out to multiple sources, ingestion functions process new documents via blob and Event Hubs triggers.
MCP binding extension exposes the retrieval function as an MCP tool, so AI agents and Copilot can call it directly
Durable Functions orchestrates multi-step retrieval workflows, including approve-before-index, parallel ingestion with fan-out, andandlong-running document processing with human-in-the-loop review.
Serverless agents runtime wraps the whole thing in an agent that decides which data sources to query based on the user’s intent, using the retrieval function as one of its tools

APIM sits in front of the Azure OpenAI calls regardless of which pattern makes the LLM request. The function app’s managed identity authenticates to both APIM and the downstream Azure services. Application Insights provides a unified view across all four patterns.

The spectrum from the Durable Functions post applies here too: RAG retrieval is mostly directed (the function knows which sources to query), but the agent layer that decides what to retrieve and how to use the results is autonomous. Azure Functions provides the event-driven, scalable retrieval infrastructure for both ends of that spectrum.

What to know before building

Choose your vector store before choosing your binding. The Azure OpenAI binding extension integrates cleanly with Azure AI Search. If you are using Cosmos DB for MongoDB vCore, a Postgres pgvector extension, or an external vector database, you write the retrieval logic directly using the SDK — the binding handles the LLM call, not the retrieval.

The embeddings binding and the chat completion binding have different cost profiles. Embeddings are cheap per token; completions are expensive. The ingestion pipeline (embeddings) can run at high volume with low cost. The retrieval + completion path (chat) is where token costs accumulate. Design your APIM quota policies accordingly.

Stateful chat sessions need external storage. The chat completion binding stores session history in Azure Table Storage by default. For multi-tenant RAG, partition session keys by user or tenant — the default key scheme does not enforce isolation automatically.

Retrieval quality determines output quality. Azure Functions handles the retrieval infrastructure; what you retrieve is your responsibility. This distinction matters more than it might seem.

With keyword search alone, a query like “List all Metallica albums” retrieves documents based on term frequency across the index. An index containing Radiohead (48 releases) and Porcupine Tree alongside Metallica will surface Radiohead and Porcupine Tree documents ahead of Metallica ones, because those bands have more indexed content matching common terms. The LLM then filters the irrelevant context and may miss albums that simply did not appear in the retrieved passages.

Hybrid search combining keyword retrieval with vector retrieval addresses this directly. Generate an embedding for the user query, add a VectorizedQuery targeting the content_vector field, and Azure AI Search fuses both signals using reciprocal rank fusion. Semantically relevant documents surface regardless of term frequency:

			
var vectorQuery = new VectorizedQuery(queryVector)
{
    KNearestNeighborsCount = 20,
    Fields = { "content_vector" }
};
var searchOptions = new SearchOptions
{
    Size = 20,
    VectorSearch = new VectorSearchOptions { Queries = { vectorQuery } }
};
// Hybrid: keyword query string + vector query sent together
var results = await searchClient.SearchAsync<MusicDocument>(
    userMessage, searchOptions, cancellationToken);

		

The companion repo implements hybrid search in MusicChatAgent.cs. Chunk size, embedding model choice, and whether you use keyword, semantic, or hybrid search have more impact on output quality than any infrastructure decision.

Testing the RAG pipeline

The companion repo supports several query patterns that exercise different aspects of the pipeline. These are worth running after ingesting a few bands to verify both the retrieval and the grounding behaviour.

Artist-specific ranking — tests hybrid search surfacing the right band:

			
{"message": "List all Metallica albums with their ratings, ordered highest to lowest"}

Date filtering via LLM reasoning — the index has no date filter; the model reasons over retrieved content:

{"message": "What albums did Tool release in the 1990s?"}

Cross-band comparison — tests whether retrieval surfaces relevant documents from multiple bands:

			
{"message": "Compare Opeth and Porcupine Tree — which has more critically acclaimed albums?"}

Rating threshold query:

{"message": "Which albums across all bands have a rating above 4.5?"}

Not-in-index grounding test — the agent should say it does not have information rather than hallucinate:

{"message": "Tell me about a band called Coldplay"}

Validation — should return HTTP 400:

{"message": ""}

The hybrid search implementation means artist-specific queries return only that artist’s documents. With keyword-only search, a query for Metallica in an index containing Radiohead (48 releases) would return mostly Radiohead documents — the vector component ensures semantic relevance wins over term frequency.

The companion repo — Music RAG Agent

The music-rag-agent companion repo implements the full two-pipeline RAG system described in this post. It scrapes band and album data from SputnikMusic, generates embeddings with the Azure OpenAI SDK, indexes into Azure AI Search, and exposes an HTTP chat endpoint that retrieves relevant passages and returns grounded answers.

After ingesting Tool, Opeth, Porcupine Tree, and Radiohead you can ask questions like:

“Which band has the highest rated albums — Tool, Opeth, Porcupine Tree or Radiohead?”

The retrieval correctly surfaces Sputnik ratings and vote counts for each album. The ingestion pipeline handles special characters in album titles (Ænima, Opiate²) via URL-safe Base64 document keys, and the EventGrid trigger ensures reliable delivery with 24-hour retry backoff.

Known Sputnik artist IDs: Tool = 83, Opeth = 932, Porcupine Tree = 328, Radiohead = 86.

Series wrap-up

This post completes the series on AI and Azure Functions. The five posts covered:

The four AI-enabled patterns and how to choose between them
Hosting MCP servers — binding extension (GA), self-hosted SDK (preview), and queue-based tools
The serverless agents runtime — markdown agents, the three configuration files, and dynamic workflows
Durable Functions for directed agentic workflows — five patterns with working C# samples verified on Azure
RAG pipelines — event-driven retrieval, the Azure OpenAI binding extension, and APIM governance

The companion code repositories are at github.com/steefjan1.

Agentic AI Design Patterns on Azure

Posted on August 15, 2026 by steefjan1970

Like a lot of people on LinkedIn right now, I have seen no shortage of infographics laying out agentic AI design patterns: tidy boxes and arrows for Tool Use, ReAct, Reflection, Planning, Orchestrator, Sequential Chain, Parallel Fan-out/Fan-in, Hierarchical, and P2P Mesh. These diagrams are a fine way to get the shape of an idea across. However, a diagram cannot tell you whether the thing actually works, or what breaks when you try to stand it up for real. So I spent the last stretch actually building agentic AI on Azure: all nine patterns, for real, on Azure’s PaaS offerings. That meant writing Bicep IaC, running azd up, making actual HTTP calls against the deployed endpoints, and running actual Application Insights queries when things went wrong. In the end I had nine independently deployable reference samples, not nine slides.

Writing the Bicep was the easy part. Getting all nine actually to stand up and respond correctly was where the real lessons were. This is a retrospective on what broke, why, and what I’d tell someone else to do to avoid it.

Lesson 1: model availability is regional, and the error message won’t always tell you clearly

The first failure mode hit me repeatedly, across multiple patterns. azd up would warn that gpt-4.1 “was not found” in the target region, then fail validation outright with InvalidResourceProperties: The specified SKU 'Standard' for model 'gpt-4.1' is not supported in this region.

This is not a bug in the Bicep. It is genuinely true that not every Azure OpenAI model and SKU combination is available in every region, and availability also varies by subscription. centralus and westeurope both rejected gpt-4.1 for me at different points, while swedencentral consistently worked across every pattern in the repo. So if you are scripting or templating Azure OpenAI deployments, do not hardcode a region and assume it will work everywhere. Instead, treat azd env set AZURE_LOCATION swedencentral as a reflexive first move whenever a fresh pattern’s deploy attempt fails.

Lesson 2: Logic Apps Standard is powerful, and standing it up correctly is its own skill

The Sequential Chain pattern (input → extract → draft → validate → output, a fixed pipeline) is a textbook fit for a low-code workflow engine. So I built it on Logic Apps Standard first, and it ended up costing me, by a wide margin, the most debugging time of the whole project. Here is what went wrong, roughly in order:

ServiceProvider-type triggers cannot have a recurrence property. If you copy a recurrence block over from an ApiConnection-style trigger example, you get a WorkflowProcessingFailed error, and the message does not obviously point at the fix.
Connector parameter names are not always what you would guess. Azure literally names the built-in OpenAI connector’s connection parameter openAIEndpoint, not endpoint. I only discovered this from Azure’s own runtime error message, not by guessing at the schema.
Windows versus Linux hosting changes what app settings you need, and not symmetrically. Windows-hosted Premium plans require WEBSITE_CONTENTAZUREFILECONNECTIONSTRING and WEBSITE_CONTENTSHARE, but carrying those same settings over to a Linux plan actively breaks content sync.
WorkflowStandard (WS1) is a hard requirement, not a suggestion. Azure explicitly rejects an ElasticPremium (EP1) substitution with “Logic Apps can be deployed only to ‘WorkflowStandard’ Sku or App Service Environments,” even though the two SKUs share the same underlying compute family.
Regional SKU quota is real, and it will not show up until you deploy. SubscriptionIsOverQuotaForSku for “WS1 VMs” hit me in two different regions. I eventually fixed it by decoupling the Logic App’s region from the rest of the pattern’s resources through a separate Bicep parameter, which let it land somewhere with quota headroom.
Deterministic role-assignment names collide across resource recreation. If you name a role assignment with guid(scope, resourceId, roleName) and then delete and recreate the resource whose managed identity receives that role, Azure refuses to update the existing assignment to point at the new principal ID. So you first have to find and delete the stale assignment by its exact resource ID.
Finally, after fixing all of the above, I ran into a Logic App instance that had simply wedged itself: Sequence contains no elements on startup, Kudu unreachable or crawling, and no combination of RBAC or config fixes helping. The only real fix was deleting and recreating the whole site resource.

Durable Functions

After all that, I rebuilt the same pattern on Durable Functions instead. An orchestrator calls three sequential activities, each a plain Azure OpenAI chat completion call, and the built-in RetryOptions does the retry work that Logic Apps’ declarative retry policies would otherwise have done. It deployed in under two minutes and worked on the first real test. The conceptual case for a low-code workflow engine still holds here, since a fixed sequence of stages is exactly what that kind of tool is for. However, if you want to move fast while building agentic AI on Azure, and you already have a working Durable Functions convention elsewhere in your stack, do not underestimate how much extra surface area Logic Apps Standard adds compared with staying in code you already know how to debug.

Lesson 3: RBAC is not one system, and Cosmos DB has its own

The P2P Mesh pattern (three Azure Functions reacting to each other’s events through Event Grid, with Cosmos DB tracking completion state) threw a 403 Forbidden: "Request blocked by Auth... does not have required RBAC permissions to perform action Microsoft.DocumentDB/databaseAccounts/readMetadata". This happened even though the function app’s managed identity already had every role I expected under Microsoft.Authorization/roleAssignments.

Here is the catch: Cosmos DB runs its own, separate, native RBAC system. A normal Azure role assignment against a Cosmos account grants control-plane access, but it does nothing for data-plane operations like reading or writing items. Instead, you need a Microsoft.DocumentDB/databaseAccounts/sqlRoleAssignments resource, scoped specifically to one of Cosmos’s own built-in role definitions. (The “Cosmos DB Built-in Data Contributor” role has a fixed, well-known GUID: 00000000-0000-0000-0000-000000000002.) So if your Bicep only includes the generic role-assignment resource type pointed at a Cosmos account, it silently grants nothing useful.

(Once I fixed that, the very next error was a 404 reading “Owner resource does not exist” on the database and container the app code expected. Provisioning the Cosmos account is not the same as provisioning the database and container inside it. Remember that if your Bicep stops at the account level.)

Lesson 4: pick the “native” endpoint type when Azure offers one

Still on P2P Mesh: wiring Event Grid subscriptions to Azure Functions endpoints through a raw webhook URL (https://<app>.azurewebsites.net/runtime/webhooks/eventgrid?functionName=X&code=<key>) produced a persistent 401 Unauthorized, Webhook endpoint validation failed. It survived two fixes I tried first: swapping the master key for the eventgrid_extension system key, and then directly inspecting the deployed function, which confirmed its eventGridTrigger binding matched what Event Grid expected. Neither fix mattered, because a plain authenticated GET against that same URL came back 400 Bad Request. That proved the key itself was fine, so the actual problem had to be something specific to Event Grid’s own validation handshake against a webhook-typed endpoint.

The actual fix was switching az eventgrid event-subscription create from a raw webhook URL to --endpoint-type azurefunction --endpoint <function-resource-id>. That is the native destination type Event Grid offers for Azure Functions, and it authenticates through the ARM resource ID instead of a function key embedded in a URL. It worked immediately. So the broader lesson: if a platform offers a first-class integration type for a specific target, prefer it over hand-rolling the equivalent with a generic webhook. That way, you inherit the platform’s own handling of edge cases you would not think of yourself.

Lesson 5: “Azure AI Foundry” isn’t one resource type

The Orchestrator pattern, a central agent hosted on Azure AI Foundry Agent Service that delegates to specialist tools, failed with a raw DNS error: Name or service not known, against a hostname the app was trying to reach. This was not an auth failure or a permissions failure. The hostname genuinely did not exist.

The root cause: the Bicep provisioned the older Azure AI Foundry resource model, a Microsoft.MachineLearningServices/workspaces hub plus a project workspace, which is the original Azure AI Studio approach. Meanwhile, the app code used Azure.AI.Projects.AIProjectClient, which only understands the newer, unified Foundry resource model: a Microsoft.CognitiveServices/accounts resource with kind: 'AIServices' and allowProjectManagement: true, plus a projects child resource. That newer model is reachable at a https://<account>.services.ai.azure.com/api/projects/<project> URL that only exists for that resource type. So two things share the “Azure AI Foundry” name, yet neither is a drop-in substitute for the other from an SDK’s point of view. As a result, it is worth explicitly checking which resource model your SDK version actually expects before wiring up IaC for it, because the error you get when you guess wrong will not obviously point at “you deployed the wrong kind of resource.”

The smaller stuff that added up

A few recurring, less dramatic gotchas worth a line each:

Double-check the resource group you think you are querying. More than once I pulled exceptions from the wrong pattern’s Application Insights instance, either because I was still in the wrong working directory, or because I grabbed the first App Insights component Azure CLI happened to return rather than the one for the resource group I actually cared about. So always sanity-check the resource group name before trusting a KQL result.
A previous session’s PowerShell variable can look like a fresh result. If a command throws partway through an assignment, the variable keeps its old value. Then, if that old value happens to look plausible (in my case, a valid-looking Durable Functions status payload from an entirely different pattern), it becomes easy to mistake it for real output from the command you just ran.
“No results” and “the backing store does not exist yet” are different failure modes. Code that gracefully handles an empty index or table does not automatically handle a missing one. For example, one specialist service in the Orchestrator pattern caught SqlException broadly and degraded gracefully when its sample table lacked seed data. By contrast, the AI Search-backed specialist had no equivalent guard for a 404 on a not-yet-created index, so it took the whole request down with it. Therefore, if you document “unseeded stores degrade gracefully,” actually test the unseeded case for every specialist, not just the one you happened to build first.

The takeaway: what building agentic AI on Azure actually requires

None of these were exotic failures. In fact, every one of them turned out to be a documented, well-understood Azure behavior once I found the right doc page. What made them expensive was that the error messages, on their own, rarely pointed straight at the fix: a DNS failure that was actually a resource-model mismatch, a 401 that was actually about endpoint type rather than credentials, and a 403 that was actually a second, unrelated RBAC system. So the practical lesson is not “Azure is fragile.” Instead, building agentic AI on Azure means dealing with more independently moving parts than the getting-started docs suggest, and the fastest way through is treating every unexplained error as worth one layer deeper of investigation before you assume the obvious fix is the real one.

I have now deployed and tested all nine patterns end to end. If you want the full breakdown of each one, including what it is for, the Azure architecture, and when to reach for it, start with the design patterns overview and go from there. The repo itself is public, so you can see exactly what each pattern provisions.

The Four Things Naive RAG Diagrams Leave Out

Posted on August 12, 2026 by steefjan1970

You might have seen the diagrams like four boxes, left to right, with indexing, retrieval, augmentation, and generation. Parse the PDF, chunk the text, embed the chunks, store the vectors. Then embed the question, search, stuff the results into a prompt, and generate. I have seen it circulating every few weeks with a fresh coat of branding and a caption promising an end to hallucination.

The diagram is not wrong. It is a decent first explanation of naive RAG. The problem, however, starts when someone treats it as a design. TThe version I saw recently ended its augmentation box with three words: zero hallucination guaranteed.

That claim is where I want to start, because it is the tell. Retrieval-augmented generation reduces fabrication. It does not eliminate it. Anyone promising zero has not yet run an evaluation against their own system.

So here are the four things the four-box picture leaves out, in the order they will hurt you. I have put runnable samples for each one in a companion repository.

Gap 1: Retrieval is not the same thing as vector search

Naive RAG diagrams draw a single arrow from question to embedding to vector database. That works beautifully in demos, because demo questions are written in the same register as the source documents.

Production questions are not. They contain product codes, policy numbers, abbreviations, proper nouns, and negations. Embeddings capture meaning, and a product code has no meaning to capture.

I built a small corpus to measure this rather than assert it: eight synthetic Dutch policy documents, two consecutive years of the same policy, a collective variant with a structurally identical pricing table under different codes, and a separate reglement for medical aids. Thirty-three chunks. Then eleven questions with a known correct chunk for each.

What vector-only missed

Vector-only retrieval got seven of the eleven right. The failures were not random:

Which discount applies to code BAS-VR-400? returned the document’s changes section, which names the code but never prices it. Right document, wrong section, and the retrieved chunk looks relevant enough to answer from.
Does medical acceptance apply to package AANV-CO-03 in 2026? returned the 2025 document, which says the opposite: right topic, wrong year, inverted answer.
How many physiotherapy treatments are in the Extra package in 2025? returned a chunk from the basic policy entirely.

That second one is the one that should worry you. It is not a near miss. The prose in the two years is nearly identical, the answer is reversed, and nothing downstream can tell. An assessor reading a fluent, cited, confidently wrong answer about acceptance criteria has no signal that anything went sideways.

Keyword search, meanwhile, nails the code lookups. Okapi BM25 has been solving this problem since before any of us had an opinion about transformers.

Worth being precise about what this does and does not prove. My first version of this corpus had three documents and eight chunks, and vector-only scored four out of five, because with eight chunks there is nothing to confuse. The gap only appears once the corpus contains things that genuinely resemble each other. If your own evaluation shows dense retrieval doing fine, check whether your test set is hard before concluding your pipeline is.

Fusion widens the pool, reranking picks the answer

Therefore, the answer is not to pick a side. Azure AI Search will run both and fuse the result lists with reciprocal rank fusion. Then a semantic reranker reorders the fused list using a cross-encoder that actually reads the query against each candidate.

Here is where my expectations were wrong, and where the measurement earned its keep. Across the eleven questions:

Strategy	Top-1 correct	MRR@5
Vector-only	7 / 11	0.77
Hybrid (BM25 + vector, RRF)	6 / 11	0.72
Hybrid + semantic reranker	11 / 11	1.00

Adding keyword search made it worse. Hybrid lost a case that vector-only got right, and fixed none.

Why fusion alone went backwards

The reason is visible in the failures. BM25 matches the literal string BAS-VR-350, and that code appears in the document’s changes section, which names codes without pricing them. Lexical matching therefore promoted chunks that contain the code and cannot answer the question. Reciprocal rank fusion then faithfully merged two ranked lists, because RRF has no notion of whether a chunk answers anything. It fuses positions, not relevance.

The cross-encoder is what fixed it. It reads the question against each candidate and understands that a question about a discount needs the row with a price in it, not the sentence announcing that the code exists. That took the same candidate set from six correct to eleven.

So the lesson is sharper than “use hybrid search”. Hybrid retrieval widens the candidate pool; reranking is what converts a wider pool into better answers. Ship the first without the second, and you may go backward quietly, because nothing in the pipeline reports that it happened.

One clean question per turn is an assumption

Query handling is the other half of this gap. The diagram assumes one clean question per turn. Real questions arrive compound: we switched to the Compleet package in March, does my son’s dental work fall under that or under the basic policy, and does the deductible apply? That is three questions. Embed the whole sentence, and you retrieve the average of three intents, which is nothing in particular.

Agentic retrieval in Azure AI Search handles that by decomposing the query into subqueries, running them in parallel, reranking each, and merging. Extractive retrieval went generally available in API version 2026-04-01. Query planning and answer synthesis remain preview. Worth knowing which half you are depending on before you promise it to a steering committee.

Gap 2: Chunking is most of the work

“Chunk text for sharp recall.” One bullet. In practice, this single decision determines more of your answer quality than your choice of model.

Fixed-size splitting is what every quickstart does and what almost nothing should do. Run a 400-character window with 50 characters of overlap over a document containing a pricing table, and the splitter lands mid-row. This is the actual output from the sample, not an illustration:

                        | EUR 3,00          | BAS-VR-100  |
| EUR 200                 | EUR 6,50          | BAS-VR-200  |
| EUR 300                 | EUR 10,00         | BAS-VR-300  |
| EUR 400                 | EUR 14,00         | BAS-VR-400  |
| EUR 500                 | EUR 19,00         | BAS-VR-500  |

## 2. Fysiotherapie

Fysiotherapie wordt vanaf de 21e

Look at what survived. No header row, so nothing says which column is the deductible, which is the monthly discount, and which is the product code. The first row is cut mid-cell: its deductible tier is gone, leaving a discount attached to nothing. No document title, so nothing says this is the 2026 basic policy rather than the 2025 one or the collective variant, all three of which carry a table of exactly this shape with different numbers. And the chunk runs on into an unrelated section about physiotherapy, ending mid-sentence.

Retrieve that, and the model has to guess. It will guess. It will sound certain. And the citation attached to it will make the wrong answer more credible, not less.

What structure-aware chunking does differently

The sample runs three chunkers over the same document and counts how many chunks ended up holding table rows with no header. Fixed-size produces one out of three. Recursive paragraph splitting produces none, but leaves every chunk without a section heading. Structure-aware produces four chunks, none headerless, none context-free.

The difference is three rules, and none of them is clever: split on headings rather than character counts, never split a table, and prepend the document title and section heading to every chunk so an isolated chunk still says what it is.

That last rule is what makes the three near-identical pricing tables in this corpus distinguishable at all. Without it, retrieval has to tell them apart on the numbers alone.

Gap 3: Retrieval without authorization is a breach with a chat interface

This is the gap that should worry you most, and it is absent from every version of the diagram I have seen.

Put every document in one index. Wire up a chat interface. Now every user can reach every document, because semantic search does not know about your authorization model. The retrieval layer will happily surface an internal work instruction, an HR file, or a legal memo to whoever asks a question shaped roughly like its contents.

The filter is a query construct, not a prompt instruction

Two things follow. First, the filter is a server-side query construct, not a prompt instruction. Telling the model “only use documents the user may see” is not a control; it is a suggestion to a system that has already been handed the text. Second, the filter must derive from validated token claims, never from anything the user typed.

Two mechanisms, and the one that fails open

Azure AI Search gives you two mechanisms. The durable one is an explicit filterable collection of group identifiers on each document plus an OData filter built from the caller’s claims, which works today on the stable API and which you own end to end. The managed one ingests RBAC scopes, ACLs, or Purview sensitivity labels alongside the content and enforces them at query time when you pass the user’s token in the x-ms-query-source-authorization header.

The managed route has a sharp edge worth memorizing. If the knowledge source was created without ingestionPermissionOptions, the index holds no permission metadata, and results come back unfiltered regardless of the header. It fails open quietly, and the only way to fix it is to recreate the knowledge source. As of the current GA release, document-level permissions on indexed sources remain in preview.

Whichever you choose, write the leak test. The sample repository includes one: a query, an unauthorized caller, and an assertion that fails the build if the restricted document comes back. Twelve lines. Run it in CI.

Gap 4: “Zero hallucination” is a claim, and claims get measured

Grounding the prompt does not guarantee a grounded answer. Three failure modes survive the diagram intact.

The model can prefer what it already knows over what you retrieved. Ask about a monthly premium that appears nowhere in your corpus, and a model trained on the open internet has plausible Dutch premiums available. It will produce one.

The model can blend two chunks into a claim neither of them makes. This is the subtle one, because every individual fact traces back to a source.

And the model can answer confidently when retrieval returned nothing relevant at all, because nothing in the naive RAG pipeline tells it that “I do not know” is an available output.

Three rules that make it measurable

Consequently, the fix is threefold and unexciting: an explicit refusal string in the prompt so refusal is detectable rather than inferred, mandatory citation of a reference identifier after every claim so each statement is checkable, and an evaluation set that contains questions your corpus cannot answer.

That last point is the one most teams skip. Everyone builds a golden set of questions the documents answer well. Almost nobody includes the withdrawn product code, the topic that was never documented, or the answer that lives in a file the user may not see. Those are exactly the cases that generate the incident report.

Score retrieval and generation separately, too. A wrong answer with good retrieval is a generation problem. A wrong answer with bad retrieval is a retrieval problem. Without both numbers, you will spend a week tuning the wrong half of the system.

What the numbers actually said

Then the result, which surprised me: fourteen out of fourteen on groundedness and valid citations, and five out of five refusals on the unanswerable questions. The model corrected false premises rather than accepting them; asked whether medical acceptance applied in 2026, it answered yes and cited; asked about a product code withdrawn before the corpus begins, it refused outright rather than interpolating a plausible price from the neighboring rows.

I want to be careful about what that number means, because it is easy to oversell in the other direction. Retrieval hit fourteen out of fourteen in the same run, so generation was working from good material throughout. This is not evidence that hallucination is solved. It is evidence that the three unexciting mechanisms above are sufficient when retrieval is doing its job, which is the argument for building all four gaps rather than any one of them.

One caveat I would want stated if someone showed me this number: the judge was the same model as the generator. A model scoring its own output shares its own blind spots, and the groundedness figure is inflated to an unknown degree by that. Use a different model for judging if the number needs to carry weight.

Where this is the wrong answer

If your corpus is fifty pages of prose, with no product codes, no tables, one audience, and low stakes, then the four-box diagram is enough. Hybrid retrieval, structure-aware chunking, security trimming, and a groundedness harness are all overhead you do not need yet. Build the simple thing, ship it, and see what breaks.

The four gaps become urgent at specific, recognizable moments: the first exact-match question that returns the wrong table row, the first document that should not be visible to everyone, and the first stakeholder who asks how you know the answers are right. If none of those have happened, you are fine.

The part the diagram gets right

Context beats prompt engineering. That much is true, and it is the reason the picture keeps circulating. But “give the model the right information at the right time” restates the problem. It is not a solution. The right information depends on hybrid retrieval and reranking. The right time depends on query decomposition. Whether the user was entitled to that information depends on trimming. And whether the model actually used it depends on evaluation.

Do all four and the numbers hold up — mine did. Skip any one of them, and you will not find out which one you skipped until someone asks a question about last year’s policy and gets this year’s answer.

Four boxes, four gaps. The gaps are where the engineering lives.

The samples are at steefjan1/naive-rag-gap: hybrid retrieval and reranking, chunking, security trimming, and groundedness evaluation, provisioned with a single azd up. Four of the five run end to end against a live service; the agentic retrieval sample needs a knowledge base that none of the scripts create, so treat that one as a sketch rather than a worked example.

They target Azure AI Search API versions current as of August 2026. Agentic retrieval and document-level permissions are both moving quickly, so check the docs before assuming a preview flag is still a preview flag.

Multi-Agent Patterns in Azure Logic Apps: Handoffs, Orchestrators, and Sequential Loops

Posted on June 22, 2026 by steefjan1970

Part 5 of 7 in the Logic Apps Agent Loop series

Part 4 covered the three tooling layers available to an Azure Logic Apps agent. A single agent with well-defined tools handles a wide range of integration scenarios, but some workloads are too complex for one agent to handle well. Azure Logic Apps multi-agent patterns let you compose multiple agent loops into a coordinated system, where each agent has a single focused responsibility and the output of one feeds directly into the next. This post covers the four patterns Microsoft has defined and includes a working demo that builds a two-agent sequential loop.

This post covers the four patterns Microsoft has defined for multi-agent composition in Azure Logic Apps: prompt chaining, routing, handoff, and orchestrator-workers and includes a demo that builds a two-agent sequential loop: a triage agent that classifies a customer request and hands off to a specialist agent.

Why Azure Logic Apps multi-agent patterns matter

A single agent loop works well when the task is bounded and the instructions can cover every case. The problem comes when a task has multiple distinct phases that require different expertise, different tools, or different models. Packing all of that into one agent’s instructions creates a sprawling, hard-to-maintain prompt. The model has to context-switch between roles in a single loop, which degrades quality and makes the run history harder to interpret.

Multi-agent patterns solve this by giving each agent a single, clear responsibility. The agents are composed at the workflow level: one agent’s output becomes another agent’s input, and each agent can have its own model, its own tools, and its own focused instructions.

The four Azure Logic Apps multi-agent patterns explained

Microsoft’s documentation defines four patterns for multi-agent composition in Logic Apps. They are ordered by complexity.

Prompt chaining

The simplest pattern. A sequence of agent loops runs one after another, where the output of each loop becomes the input to the next. Each agent has a single focused task: extract, then format, then sort, then summarise. The chain is linear and predictable.

Use prompt chaining when the workload can be decomposed into sequential steps with clear handover points and when the output of each step is well-defined. A business report processing chain, raw data in, executive summary out, is the canonical example from the Microsoft documentation.

Routing

A classification agent examines the incoming request and routes it to one of several specialist agent loops based on what it finds. The routing agent does not do the work itself it decides which agent should do the work and passes control there.

Use routing when incoming requests fall into distinct categories that need different handling: a customer service triage agent that routes billing queries to a billing agent loop, technical questions to a technical support agent loop, and general inquiries to a general response agent loop. The routing pattern prevents optimization conflicts, allowing a billing specialist agent to be tuned for billing tasks without being distracted by technical support scenarios.

Handoff

Similar to routing but more dynamic. Instead of a central classifier making an upfront routing decision, each agent loop decides during its own execution whether it needs to hand off to another agent. The handoff preserves conversation context and state across the transition the receiving agent knows the full history of what the previous agent did and said.

Use handoff when the trigger for transferring control depends on what emerges during the conversation: a general support agent that escalates to a technical specialist when it detects a complex issue, or a research agent that hands off to a writer agent once it has gathered enough material. The handoff pattern mimics human escalation patterns: a front-line agent handles what it can and passes on what it cannot.

Orchestrator-workers

The most sophisticated pattern. A central orchestrator agent dynamically decomposes a task into subtasks and delegates each subtask to a worker agent loop. The worker agents operate as tools that the orchestrator can invoke, exactly the tool provider pattern from Part 4, applied to agents rather than connectors.

Use orchestrator-workers when you cannot predict the required subtasks in advance. A coding agent that needs to make changes to an unpredictable number of files, a research agent that gathers information from multiple dynamic sources, or a content pipeline with a writer, reviewer, and publisher working together, these are all orchestrator-worker scenarios. The orchestrator dynamically determines what needs to be done; the workers execute it.

Demo: Building a sequential agent loop — Extract and Summarise

This demo builds a two-agent prompt chaining workflow in a new sequential-agents workflow inside la-agent-loop. The scenario is a business report processing chain: Agent 1 extracts key facts and metrics from a raw text input, Agent 2 takes those facts and writes a concise executive summary. The output of Agent 1 feeds directly into Agent 2 — this is the prompt chaining pattern in its simplest form.

Prerequisites

The la-agent-loop Standard logic app from previous posts
An Azure OpenAI / Foundry Models connection already configured

Step 1: Create the workflow

In la-agent-loop, click Create and name the workflow sequential-agents. Select Autonomous Agents as the workflow type. Logic Apps creates the workflow with an HTTP trigger and an empty Agent action.

Step 2: Configure the HTTP trigger

Click the When an HTTP request is received trigger and paste this request body schema:

{ "type": "object", "properties": { "report": { "type": "string" } }, "required": ["report"] }

Step 3: Configure the Extract Agent

Click the first Agent action and rename it Extract Agent. Configure it:

AI model: your GPT-4o / Foundry Models connection
Instructions: You are a data extraction specialist. Extract all numerical values, metrics, and key facts from the provided text. Return them as a clean bulleted list. Do not summarise or interpret — only extract.
User instructions item – 1: select report from the HTTP trigger dynamic content

Step 4: Add a Compose action

This is a critical step. The Extract Agent output is a JSON object containing a messages array — not a plain string. The Summarize Agent cannot process it directly. A Compose action between the two agents extracts the plain text content.

Click + below the Extract Agent container and add Add an action → Simple Operations → Compose. Set the Inputs expression to:

outputs('Extract_Agent')?['body']?['messages'][0]['content']

This extracts the bulleted list text from the Extract Agent’s output object and passes it as a clean string to the next agent.

Step 5: Add the Summarize Agent

Click + below the Compose action and select Add an agent. Rename it Summarize Agent. Configure it:

AI model: your GPT-4o / Foundry Models connection
Instructions: You are an executive communications specialist. Take the provided list of facts and metrics and write a concise three-sentence executive summary suitable for a board report. Be professional and direct.
User instructions item – 1: select the Outputs of the Compose action from the dynamic content picker

Step 6: Add a Response action

Click + below the Summarize Agent container and add a Response action:

Status Code: 200
Content-Type header: application/json
Body: set the expression to outputs('Summarize_Agent')?['body']?['messages'][0]['content']

Step 7: Save and test

Save the workflow and POST this to the trigger URL:

{ "report": "Q3 revenue was €4.2M, up 18% year on year. Customer acquisition cost dropped to €142, down from €198. Net promoter score reached 67. Headcount grew from 43 to 51. Churn rate fell to 2.3%." }

The workflow runs in approximately 16 seconds and returns a clean executive summary:

In Q3, revenue reached €4.2M, reflecting an 18% year-on-year increase, supported by a significant reduction in customer acquisition cost from €198 to €142. The company saw operational growth with headcount rising from 43 to 51, while maintaining strong customer satisfaction, evidenced by a Net Promoter Score of 67 and a low churn rate of 2.3%. These metrics highlight sustained growth and improved efficiency across key areas.

The run history shows two distinct agent iterations, Extract Agent and Summarize Agent, each with their own Think → Observe cycle, confirming the prompt chaining pattern is working end to end.

Practitioner note: The Compose action between the two agents is not optional. Logic Apps Agent actions return a structured JSON object not a plain string, so the second agent cannot consume the first agent’s output directly from dynamic content. The Compose expression outputs('Extract_Agent')?['body']?['messages'][0]['content'] bridges this gap. This is not documented clearly by Microsoft at the time of writing and is the most common point of failure when building sequential agent loops.

Choosing the right pattern

Pattern	Complexity	Use when
Prompt chaining	Low	Sequential steps with clear handover points
Routing	Low–medium	Distinct input categories needing different handling
Handoff	Medium	Dynamic escalation based on conversation content
Orchestrator-workers	High	Unpredictable subtasks requiring dynamic decomposition

The patterns are not mutually exclusive. A production customer service system might use routing to direct initial requests, handoff for mid-conversation escalations, and prompt chaining within each specialist agent to process the request through multiple steps.

What comes next

Part 6 covers securing agentic workflows, the expanded caller surface introduced by multi-agent and conversational patterns, Easy Auth setup for production, and Managed Identity for backend connections.

Anatomy of an Agent Loop in Azure Logic Apps

Posted on June 2, 2026 by steefjan1970

Part 2 of 7 in the Logic Apps Agent Loop series

Part 1 explained why the Azure Logic Apps agent loop is a different design paradigm from conventional workflow automation. This post gets hands-on with the anatomy of that loop. We will look at the four building blocks that make up every agent loop trigger, instructions, connected model, and tools, and walk through how to wire them together in a Standard logic app.

By the end of this post you will have a working autonomous agent that accepts a prompt from a trigger, reasons over it using Azure OpenAI, invokes a connector action as a tool, and returns a result. The run history will show you exactly how the loop iterated.

The Azure Logic Apps agent loop: four building blocks

Before opening the designer, it helps to have a clear mental model of what you are assembling. Every Azure Logic Apps agent loop consists of four parts.

Trigger

The trigger starts the workflow, exactly as it does in any nonagentic Logic Apps workflow. For an autonomous agent, this can be any supported trigger an HTTP request, a timer, a Service Bus message, a new email, or anything else in the connector library. The trigger’s output becomes the initial input to the agent: the prompt or data the model will reason over.

Instructions

Instructions are the system prompt for the agent. You provide them as a block of natural language text in the agent action’s configuration pane. They define the agent’s role, what it can and cannot do, how it should respond, and any constraints it should observe. A well-written instructions block is the single most important factor in how well the agent performs. Think of it as the job description you hand to the model at the start of every run.

Connected model

The agent needs a language model to reason with. In Standard Logic Apps, you connect the agent to an Azure OpenAI Service resource and specify the model deployment to use — typically a GPT-4o deployment. The agent sends the instructions, the trigger input, and the results of any tool calls to the model at each iteration of the loop. The model’s response tells the agent what to do next.

Tools

A tool is a sequence of one or more connector actions that the agent can choose to invoke. You build tools directly in the Logic Apps designer by adding actions from the connector gallery inside the agent action. Each tool gets a name and a description — the model uses these to decide which tool to call and when. A single agent can have multiple tools. An agent with no tools can still respond to prompts using the model’s built-in knowledge, but it cannot take action on external systems.

The diagram below shows how these four parts fit together inside a single agent loop execution.

Building your first agent loop in Azure Logic Apps

The demo for this post is deliberately simple: an agent that receives a topic via an HTTP trigger, uses Azure OpenAI to generate a summary, and returns the result to the caller. One trigger, one model, one tool is enough to see all four building blocks in action and to read a meaningful run history.

Prerequisites

A Standard logic app resource deployed in Azure
An Azure OpenAI Service resource with a GPT-4o model deployment
Contributor access to both resources

Step 1: Create the workflow

In the Azure portal, open your Standard logic app and select Workflows from the sidebar. Choose Add, then select Autonomous Agents as the workflow type. Give the workflow a name and select Stateful. Logic Apps creates a new workflow with an empty agent action already in place.

Step 2: Configure the trigger

The autonomous agent workflow template starts with a When a HTTP request is received trigger by default. Leave the method as POST. In the request body JSON schema, add a single property: topic of type string. This is the input the agent will work with.

Step 3: Write the instructions

Select the agent action in the designer to open its configuration pane. On the Parameters tab, find the Instructions field. Enter something like the following:

You are a research assistant. When given a topic, use the available tools to retrieve relevant information and return a concise summary of no more than three sentences. Always cite your source.

Keep instructions specific and bounded. Vague instructions produce unpredictable behaviour. The model will take the instructions literally, so precision matters.

Step 4: Connect the model

Still on the Parameters tab, select Add connection under the model configuration section. Choose Azure OpenAI Service, select your resource, and choose your GPT-4o deployment. Logic Apps establishes the connection and stores it against the workflow.

Step 5: Add a tool

Inside the agent action, select Add a tool. This opens the connector gallery filtered to actions that can be used as tools. For this demo, add the HTTP action as a tool — name it search_web, give it the description “Retrieves content from a given URL”, and configure it to accept a URL as input. In a production scenario you would use Azure AI Search or a more capable connector here; the HTTP action keeps the demo self-contained.

Step 6: Save and run

Save the workflow. Use a REST client to POST a JSON body like {"topic": "Azure Logic Apps agent loop"} to the workflow’s trigger URL. The agent fires, the model reasons over the instructions and the topic, invokes the search tool, and returns a summary.

Reading the run history

he run history is where the Azure Logic Apps agent loop becomes visible. Open the workflow’s Run history and select the latest run. You will see the trigger, followed by the agent action. Expand the agent action and you will find each iteration of the loop shown as a numbered step: the model’s reasoning output, the tool call with its inputs and outputs, and the model’s decision on whether to loop again or return a final answer.

This is the key difference from a nonagentic run history. In a conventional workflow, the run history shows a flat list of actions. In an agent loop, it shows a nested, iterative structure the model’s chain of thought made visible.

For a simple prompt, you may see a single iteration. For a more complex task involving multiple tool calls, you will see the loop unfold across three, five, or more steps. Each step shows exactly what the model decided and why.

Standard versus Consumption: model connections

In Standard logic apps, you configure the model connection yourself — selecting an Azure OpenAI Service resource and specifying the deployment. This gives you full control over which model version you use, where it is hosted, and how it is secured via Managed Identity.

In Consumption logic apps (currently in public preview), the model connection is handled via Microsoft Foundry and the configuration is more constrained. For any production workload, Standard remains the right choice.

What comes next

The agent in this post is autonomous it runs without human interaction, triggered by an HTTP call and returning a result when done. That covers a wide range of integration scenarios, but not all of them. Some tasks require a back-and-forth with a user: a support conversation, a guided data-entry flow, a multi-turn research session.

The next part will cover exactly that distinction, autonomous versus conversational agentic workflows, and walk through when to choose each pattern and what changes in the designer when you do.

Why the Agent Loop Changes Everything in Azure Logic Apps

Posted on May 28, 2026 by steefjan1970

Part 1 of 7 in the Logic Apps Agent Loop series

The Azure Logic Apps agent loop introduces a fundamentally different way to design workflows on the platform. While conventional Logic Apps workflows follow a fixed sequence of steps defined at design time, the agent loop delegates reasoning to a large language model at runtime, looping through think, act, and observe cycles until a task is complete. This post opens a seven-part series on building agentic workflows in Logic Apps. It starts with the question that matters most: why does this change anything?

For years, Azure Logic Apps has been the platform of choice for integration architects who need to orchestrate business processes across cloud services and on-premises systems. You build a workflow, wire up triggers and actions, define your conditions, handle your errors, deploy, and move on. The flow is predictable (deterministic): given the same inputs, it does the same thing every time. That predictability is the point.

The agent loop breaks that contract, deliberately and usefully.

With the introduction of agentic workflows in Azure Logic Apps, Microsoft has extended the platform from a fixed automation engine into something that can reason, adapt, and decide. At its core, the agent loop drives this shift. It is a repeating process: the connected language model thinks through a problem, selects a tool, acts on the result, and decides whether the task is done.Unlike a conventional workflow, there is no hardcoded sequence of steps. Instead, the model determines the path based on the task.

This post is the opening of a seven-part series on building agentic workflows in Azure Logic Apps. Before going hands-on with triggers, connectors, and multi-agent patterns in later posts, this one makes the case for why the agent loop matters and what it fundamentally changes about how you think about workflow design.

How the Azure Logic Apps agent loop differs from nonagentic workflows

Nonagentic Logic Apps workflows are excellent at exactly the kind of work they were designed for: stable, predictable, repeatable processes. An approval workflow, an ETL pipeline, and a B2B message exchange are all scenarios where the path through the workflow is known in advance. The trigger fires, the conditions evaluate, the actions execute in sequence, and the run history tells you exactly what happened and why.

The challenge arises when the environment you are integrating with is unstable or unpredictable. When incoming data is unstructured. Or when the right action depends on context that cannot be captured in a condition expression. Or when you need to handle a customer query that could go a dozen different directions depending on what the customer actually says.

These are the cases where deterministic workflows buckle. You end up building sprawling switch-case structures, hardcoding edge cases as branches, and constantly patching the workflow every time a new variation appears. The workflow becomes a maintenance problem rather than a solution.

Agentic workflows excel in dynamic environments where unexpected events occur, the choice of the right tool relies on the input, and the system must manage unstructured data without specific instructions for each variant.

The agent loop: Think, Act, Learn

How the agent loop works: Think, Act, Learn

The Azure Logic Apps agent loop follows a three-step process.

Think. The agent collects available information: task instructions, prior inputs, and previous tool results. It then passes all of this to the connected language model.The model reasons over the context and decides what to do next: invoke a tool, ask a follow-up question, or return a final answer.

Act. In Logic Apps, tools are actions drawn from 400+ connectors. These include Azure OpenAI, Azure AI Search, Office 365, and custom APIs. Once the action runs, the result feeds back into the next cycle.

Optionally, the loop adapts. The agent can use feedback or external signals to adjust its behaviour over time, though this is the most advanced capability and not required for most workflows.

Iterations, not instructions

This loop continues think, act, observe, decide until the model determines the task is complete. You can change the number of iterations as needed. A simple query might resolve in one loop. A complex multi-step task might require five or ten.

The diagram below shows the difference between a conventional non-agentic workflow, which follows a linear sequence of predetermined steps, and the agent loop, which dynamically iterates until the model determines that the task is complete.

Agent versus nonagentic: a structural comparison

The difference is not just philosophical. It shows up in how you design, deploy, and maintain the workflow.

In a nonagentic workflow, the logic architect owns the decision tree. Every branch, every condition, every action path is explicitly modelled. This is powerful for known, bounded scenarios, but it places all the reasoning burden on the architect at design time.

In an agentic workflow, the reasoning is delegated to the model at runtime. The architect’s job shifts: instead of modelling every path, you define the agent’s instructions, give it the right tools, and trust the model to navigate the task. This is a different skill and a different mindset closer to prompt engineering and system design than to traditional workflow modelling.

The Microsoft documentation puts it plainly: agentic workflows can adapt to environments where unexpected events happen, choose which tools to use based on prompts and available data, and handle unstructured data at a level of flexibility that nonagentic workflows simply cannot match. Moreover, nonagentic workflows function best in stable environments with static, predictable, repetitive tasks.

Neither is universally better. They address different problems. But for integration architects, the arrival of the agent loop means Logic Apps can now cover territory that previously required a custom-coded application or a fully separate agent framework.

Standard versus Consumption: what you need to know now

Azure Logic Apps offers two hosting models: Standard (single-tenant, runs on Azure Functions runtime) and Consumption (multitenant, pay-per-execution). Agentic workflows are fully available in Standard. Consumption support is in public preview as of early 2026 and carries some restrictions.

For production agentic workloads, Standard is the right choice today. The rest of this series will use Standard throughout, with notes where the Consumption behaviour differs.

What this series covers

The seven posts in this series move from concept to production:

Why the agent loop changes everything — this post
Anatomy of an agent loop — instructions, the connected model, tool calls, and how the loop iterates
Autonomous versus conversational workflows — choosing between unattended execution and human-in-the-loop patterns
Building tools for the agent — connectors, custom connectors, and MCP servers as tool providers
Multi-agent patterns — handoffs, orchestrators, and sequential agent loops
Securing agentic workflows — authentication, the expanded caller surface, and Easy Auth
Observability, pricing, and running in production — Application Insights, agent loop pricing, and DevOps deployment

The next post gets hands-on: we will look at the anatomy of a single agent loop in the Logic Apps designer, walk through the instructions pane, wire up Azure OpenAI as the model, and watch the run history to see how the iterations unfold.

Cloud Perspectives

Steef-Jan Wiggers

Tag Archives: Azure OpenAI