This is the fifth and final post in the series on AI and Azure Functions. The first post mapped all four AI-enabled patterns. Posts two through four went deep on MCP server hosting, the serverless agents runtime, and Durable Functions for directed agentic workflows. This post covers the fourth pattern: retrieval-augmented generation.
RAG is where Azure Functions earns its place through a capability that is easy to underestimate: the ability to handle multiple events from multiple data sources simultaneously. A RAG pipeline is only as good as its retrieval layer, and retrieval latency in production comes down to two things: how fast you can query your data sources, and how fast you can scale when demand spikes. Azure Functions on Flex Consumption handles both.
What RAG on Azure Functions actually looks like
The Microsoft Learn documentation is deliberately brief on this pattern: “RAG systems require fast data retrieval and processing. Functions can interact with multiple data sources simultaneously and provide the rapid scale required by RAG scenarios.”
That one-liner contains three architectural decisions worth unpacking.
Multiple data sources simultaneously. A retrieval layer that queries one source at a time is a bottleneck. A function app can fan out — trigger parallel calls to Azure AI Search, a Cosmos DB vector store, a blob-indexed knowledge base, and a case history API — collect all results, and pass the aggregated context to the language model in a single call. This is the fan-out/fan-in pattern from the Durable Functions post, applied to retrieval rather than to a directed workflow.
Event-driven. The most common RAG pattern in tutorials is synchronous: the user sends a query, retrieval runs, and the LLM responds. But production RAG often has an asynchronous dimension — new documents arrive, get processed, get indexed. Azure Functions handles both sides: HTTP-triggered retrieval for the synchronous path, and blob/Event Hubs/Cosmos DB change feed triggers for the ingestion pipeline that keeps the index current. The same function app can serve both.
Rapid event-driven scaling. The Azure OpenAI binding extension adds stateful chat session support — the function maintains conversation history across turns without you writing session management code. Combined with Flex Consumption’s scale-to-zero billing, that means a RAG-enabled chat endpoint costs nothing when idle and scales in seconds when traffic arrives.
The Azure OpenAI binding extension
The binding extension is the most concrete piece of infrastructure the docs call out for RAG. It provides three bindings relevant to RAG scenarios:
Text completion input binding — calls an Azure OpenAI deployment and returns the response. Used for one-shot prompt/response patterns.
Chat completion input binding — maintains a stateful chat session across function invocations. The session history is stored externally (Azure Table Storage by default) and automatically injected into each prompt. This is what the “Custom chat bot” sample in the docs demonstrates.
Embeddings input binding — generates vector embeddings from text. Used in the ingestion pipeline to embed documents before writing them to a vector store.
A simple RAG function using the chat completion binding looks like this in C#:
[Function(nameof(RagChat))]public static async Task<IActionResult> RagChat( [HttpTrigger(AuthorizationLevel.Function, "post")] HttpRequestData req, [TextCompletion( "{query}", Model = "%OPENAI_DEPLOYMENT%", SystemPrompt = "You are a helpful assistant. Answer using only the context provided." )] TextCompletionResponse completion){ return new OkObjectResult(completion.Content);}
The binding handles the Azure OpenAI call, authentication via managed identity, and response parsing. You write the retrieval logic — what context to inject into {query} — and the binding handles the LLM call.
The production version adds a retrieval step before the binding executes. In practice, this means the HTTP trigger extracts the user query, a call to Azure AI Search (or your vector store of choice) retrieves relevant passages, those passages are injected into the prompt template, and then the binding submits the augmented prompt to the LLM.
APIM as the LLM gateway layer
The binding extension simplifies the function code, but it routes directly to Azure OpenAI — there is no governance layer between the function and the model by default. For production RAG pipelines, that matters: you need rate limiting per client, token quota management, cost attribution across tenants or use cases, and a single point for monitoring and logging LLM calls.
APIM fills that role. Place it between the function app and Azure OpenAI, configure an inbound policy to validate the request and apply per-subscription token quotas, and configure an outbound policy to log usage. The function app calls the APIM endpoint rather than the Azure OpenAI endpoint directly.
One practical constraint from real deployment experience: if you use the Azure AI Foundry Agent Service SDK alongside Azure Functions, the SDK routes LLM calls directly to Azure OpenAI and bypasses APIM entirely. The standard OpenAI SDK doesn’t behave this way—calls flow through APIM as expected. The Citadel Platform series covers this in detail, including how to structure APIM policies for the token quota patterns that matter in production.
For RAG pipelines specifically, two APIM policies are worth implementing:
Token quota by subscription key. Set a per-day or per-hour token budget for each API consumer. RAG queries are often significantly larger than simple completions because the retrieved context inflates the prompt — a query with 2,000 tokens of retrieved passages consumes far more quota than a direct LLM call. Factor this into your quota design.
Semantic caching. APIM’s semantic caching policy stores LLM responses and returns cached results for semantically similar queries. For a RAG pipeline serving a knowledge base that changes infrequently, this can dramatically reduce token consumption and latency.
The ingestion pipeline
The retrieval side of RAG gets most of the attention, but the ingestion pipeline- how documents get into the index is where Azure Functions adds less-discussed value.
When a new document arrives in blob storage, the function extracts text, generates a vector embedding, and writes the result to Azure AI Search. An Event Hubs trigger handles high-throughput ingestion where many documents arrive simultaneously. A Cosmos DB change feed trigger keeps a vector index in sync with a transactional database.
The key advantage over a dedicated ingestion service is that the same managed identity, Application Insights workspace, and azd deployment pipeline cover both retrieval and ingestion. Operational complexity stays low even as the pipeline grows.
One constraint to know before building: blob triggers on Flex Consumption require EventGrid as the source. Standard polling blob triggers are not supported on the Flex Consumption plan. This means you must configure an EventGrid system topic and event subscription after provisioning — a two-step post-provision operation that cannot be included in the same Bicep deployment as the function app.
A minimal ingestion function using the Azure OpenAI SDK for embeddings — the recommended approach until the embeddings binding extension reaches GA:
[Function(nameof(IngestDocument))]public async Task Run( [EventGridTrigger] EventGridEvent eventGridEvent, CancellationToken cancellationToken){ // Read blob from storage, generate embedding, write to Azure AI Search var blobContent = await ReadBlobAsync(eventGridEvent.Subject, cancellationToken); var embeddingClient = openAiClient.GetEmbeddingClient(embeddingsDeployment); var embedding = await embeddingClient.GenerateEmbeddingAsync( blobContent, cancellationToken: cancellationToken); var document = new SearchDocument { ContentVector = embedding.Value.ToFloats().ToArray() }; await searchClient.IndexDocumentsAsync( IndexDocumentsBatch.Upload([document]), cancellationToken: cancellationToken);}
The EventGrid trigger fires on MicrosoftStorage.BlobCreated events; the function reads the blob, generates the vector, and writes to the index. EventGrid delivery includes retry logic —failed events are retried with exponential backoff for up to 24 hours.
Connecting the four patterns
This is the final post in the series, so it is worth mapping how the four AI patterns relate to each other in a production architecture. They are not mutually exclusive.
A realistic enterprise RAG system might use all four:
- Azure Functions RAG pipeline handles retrieval — HTTP-triggered queries fan out to multiple sources, ingestion functions process new documents via blob and Event Hubs triggers.
- MCP binding extension exposes the retrieval function as an MCP tool, so AI agents and Copilot can call it directly
- Durable Functions orchestrates multi-step retrieval workflows, including approve-before-index, parallel ingestion with fan-out, andandlong-running document processing with human-in-the-loop review.
- Serverless agents runtime wraps the whole thing in an agent that decides which data sources to query based on the user’s intent, using the retrieval function as one of its tools
APIM sits in front of the Azure OpenAI calls regardless of which pattern makes the LLM request. The function app’s managed identity authenticates to both APIM and the downstream Azure services. Application Insights provides a unified view across all four patterns.
The spectrum from the Durable Functions post applies here too: RAG retrieval is mostly directed (the function knows which sources to query), but the agent layer that decides what to retrieve and how to use the results is autonomous. Azure Functions provides the event-driven, scalable retrieval infrastructure for both ends of that spectrum.
What to know before building
Choose your vector store before choosing your binding. The Azure OpenAI binding extension integrates cleanly with Azure AI Search. If you are using Cosmos DB for MongoDB vCore, a Postgres pgvector extension, or an external vector database, you write the retrieval logic directly using the SDK — the binding handles the LLM call, not the retrieval.
The embeddings binding and the chat completion binding have different cost profiles. Embeddings are cheap per token; completions are expensive. The ingestion pipeline (embeddings) can run at high volume with low cost. The retrieval + completion path (chat) is where token costs accumulate. Design your APIM quota policies accordingly.
Stateful chat sessions need external storage. The chat completion binding stores session history in Azure Table Storage by default. For multi-tenant RAG, partition session keys by user or tenant — the default key scheme does not enforce isolation automatically.
Retrieval quality determines output quality. Azure Functions handles the retrieval infrastructure; what you retrieve is your responsibility. This distinction matters more than it might seem.
With keyword search alone, a query like “List all Metallica albums” retrieves documents based on term frequency across the index. An index containing Radiohead (48 releases) and Porcupine Tree alongside Metallica will surface Radiohead and Porcupine Tree documents ahead of Metallica ones, because those bands have more indexed content matching common terms. The LLM then filters the irrelevant context and may miss albums that simply did not appear in the retrieved passages.
Hybrid search combining keyword retrieval with vector retrieval addresses this directly. Generate an embedding for the user query, add a VectorizedQuery targeting the content_vector field, and Azure AI Search fuses both signals using reciprocal rank fusion. Semantically relevant documents surface regardless of term frequency:
var vectorQuery = new VectorizedQuery(queryVector){ KNearestNeighborsCount = 20, Fields = { "content_vector" }};var searchOptions = new SearchOptions{ Size = 20, VectorSearch = new VectorSearchOptions { Queries = { vectorQuery } }};// Hybrid: keyword query string + vector query sent togethervar results = await searchClient.SearchAsync<MusicDocument>( userMessage, searchOptions, cancellationToken);
The companion repo implements hybrid search in MusicChatAgent.cs. Chunk size, embedding model choice, and whether you use keyword, semantic, or hybrid search have more impact on output quality than any infrastructure decision.
Testing the RAG pipeline
The companion repo supports several query patterns that exercise different aspects of the pipeline. These are worth running after ingesting a few bands to verify both the retrieval and the grounding behaviour.
Artist-specific ranking — tests hybrid search surfacing the right band:
{"message": "List all Metallica albums with their ratings, ordered highest to lowest"}
Date filtering via LLM reasoning — the index has no date filter; the model reasons over retrieved content:
{"message": "What albums did Tool release in the 1990s?"}
Cross-band comparison — tests whether retrieval surfaces relevant documents from multiple bands:
{"message": "Compare Opeth and Porcupine Tree — which has more critically acclaimed albums?"}
Rating threshold query:
{"message": "Which albums across all bands have a rating above 4.5?"}
Not-in-index grounding test — the agent should say it does not have information rather than hallucinate:
{"message": "Tell me about a band called Coldplay"}
Validation — should return HTTP 400:
{"message": ""}
The hybrid search implementation means artist-specific queries return only that artist’s documents. With keyword-only search, a query for Metallica in an index containing Radiohead (48 releases) would return mostly Radiohead documents — the vector component ensures semantic relevance wins over term frequency.
The companion repo — Music RAG Agent
The music-rag-agent companion repo implements the full two-pipeline RAG system described in this post. It scrapes band and album data from SputnikMusic, generates embeddings with the Azure OpenAI SDK, indexes into Azure AI Search, and exposes an HTTP chat endpoint that retrieves relevant passages and returns grounded answers.
After ingesting Tool, Opeth, Porcupine Tree, and Radiohead you can ask questions like:
“Which band has the highest rated albums — Tool, Opeth, Porcupine Tree or Radiohead?”
The retrieval correctly surfaces Sputnik ratings and vote counts for each album. The ingestion pipeline handles special characters in album titles (Ænima, Opiate²) via URL-safe Base64 document keys, and the EventGrid trigger ensures reliable delivery with 24-hour retry backoff.
Known Sputnik artist IDs: Tool = 83, Opeth = 932, Porcupine Tree = 328, Radiohead = 86.
Series wrap-up
This post completes the series on AI and Azure Functions. The five posts covered:
- The four AI-enabled patterns and how to choose between them
- Hosting MCP servers — binding extension (GA), self-hosted SDK (preview), and queue-based tools
- The serverless agents runtime — markdown agents, the three configuration files, and dynamic workflows
- Durable Functions for directed agentic workflows — five patterns with working C# samples verified on Azure
- RAG pipelines — event-driven retrieval, the Azure OpenAI binding extension, and APIM governance
The companion code repositories are at github.com/steefjan1.
