Hosting MCP Servers on Azure Functions: GA vs. Preview

The Model Context Protocol (MCP) has become the standard way AI agents and models interact with external systems. If you are building something on Azure and need to expose tools to an AI client, the question is no longer whether to use MCP; it is which hosting option to choose.

Azure Functions supports three distinct approaches. Two of them host MCP servers directly. The third uses message queues instead of MCP calls altogether. Each reflects a different set of trade-offs. This post maps them out so you can pick the right one before you write any code.

Why use Azure Functions for hosting MCP servers?

The doc makes the case concisely: Functions scales efficiently to handle demand and provides binding extensions that simplify AI integration. Both matter for MCP hosting specifically.

MCP servers are called on demand. An AI client sends a request, the server responds, and then goes quiet. That is a bursty, event-driven pattern. Flex Consumption and Elastic Premium both handle it well. Flex Consumption gives you scale-to-zero billing for bursty tool workloads. Elastic Premium gives you pre-warmed instances, predictable latency, and VNet integration for tools that need to reach internal APIs and databases.

Managed identity is the other key piece. Your MCP server likely needs to call downstream Azure services: blob storage, Cosmos DB, and Azure AI Search. With managed identity, you do not need to manage credentials inside your function code, and you do not need to pass secrets through your MCP configuration.

The three ways to host MCP servers on Azure Functions

Option 1: MCP binding extension (GA)

This is the right default for most teams. The binding extension lets you build an MCP server using the standard Functions programming model triggers, bindings, local development with Core Tools, and deployment via azd. You annotate a function with McpToolTrigger, and it is automatically exposed as an MCP tool.

The comparison table from the Microsoft Learn documentation makes the feature set clear:

FeatureMCP binding extension
Support levelGA
Programming modelFunctions, triggers and bindings
Stateful executionSupported
LanguagesC# (isolated), Python, TypeScript, JavaScript, Java
Other requirementsNone
ImplementationMCP binding extension

Stateful execution support is the binding extension’s meaningful advantage over the self-hosted SDK option. If your MCP server needs to maintain session state across tool calls, for example, a multi-turn data retrieval scenario, only the binding extension handles that today.

The quickstart template (remote-mcp-functions-dotnet) is a useful starting point. It scaffolds the project via azd init, runs locally with the Azurite storage emulator, and includes a .vscode/mcp.json file that wires the local endpoint directly into GitHub Copilot’s agent mode for testing. That local-to-Copilot loop is genuinely quick; you can test a tool call from Copilot chat without deploying anything.

Before you build: a toolchain issue to know about

There is a confirmed bug in Microsoft.Azure.Functions.Worker.Sdk (all versions to 2.0.7) where the auto-generated WorkerExtensions.csproj is hardcoded to net6.0. The MCP extension package requires net8.0, so the build fails out of the box with NU1202: Package is not compatible with net6.0. The Microsoft Learn quickstart does not mention this.

The workaround that actually works has three parts: add an extensionBundle to host.json so the host loads MCP support at runtime rather than through the build-time generator; mark all extension PackageReference entries with PrivateAssets="all" so their build targets don’t re-trigger the generator; and use 1.0.0-preview.3 rather than 1.0.0 or later preview versions, because the generated project still targets net6.0 and needs a package version it can restore against that framework. The companion repo has all three changes applied and a detailed explanation in the README.

The McpToolTrigger attribute is the core mechanism. Here is what the minimal C# version looks like from the quickstart:

[Function(nameof(SayHello))]
public string SayHello(
[McpToolTrigger(HelloToolName, HelloToolDescription)] ToolInvocationContext context
)
{
logger.LogInformation("C# MCP tool trigger function processed a request.");
return "Hello I am MCP Tool!";
}

The trigger attribute registers the function as an MCP tool and handles the protocol negotiation. You write the tool logic; the extension handles the MCP wire format.

Use this when: you are building a new MCP server and do not have an existing codebase using MCP SDKs. It is the fastest path from zero to a deployed, governed MCP server on Azure.

Avoid it when: you already have an MCP server built with an official MCP SDK and want to lift it into Azure without rewriting it in the Functions model.


Note: Microsoft has samples for this option. You can find all the MCP extension samples at aka.ms/remote-mcp

Option 2: Self-hosted MCP servers via official MCP SDKs (preview)

This option lets you take an existing MCP server built with the official MCP SDKs and host it on Azure Functions without rewriting it as a binding-extension-style function app. Functions acts as the hosting runtime; your MCP SDK code runs via custom handlers.

FeatureSelf-hosted MCP servers
Support levelPreview
Programming modelStandard MCP SDKs
Stateful executionNot currently supported
LanguagesC# (isolated), Python, TypeScript, JavaScript, Java
Other requirementsStreamable HTTP transport
ImplementationCustom handlers

Two constraints define whether this option is viable for you right now.

  • First: Streamable HTTP transport is required. The self-hosted option does not support Server-Sent Events (SSE) transport. Suppose your existing MCP server relies on SSE, which many early implementations did; you need to migrate to Streamable HTTP before hosting on Functions. That is a non-trivial change for an established codebase.
  • Second: stateful execution is not supported during preview. If your MCP server is stateless most tool servers are this is not a problem. But if you need session continuity across calls, stay with the binding extension.

The documentation also flags that configuration details for self-hosted MCP servers change during the preview period. That means operational overhead: you may need to adjust configuration as the feature evolves. Factor that into a production timeline.

The sample (Weather server) shows the self-hosted pattern working correctly for a stateless tool. It is a useful reference for understanding the custom handler wiring, but it does not represent a feature-complete production deployment yet.

Use this when: you have an existing MCP server built with official MCP SDKs, it uses Streamable HTTP transport, it is stateless, and you want to run it on Functions without adopting the binding extension model.

Avoid it when: you need stateful execution, you rely on SSE transport, or you are building from scratch. In those cases, the binding extension is the better path.

Note: There is another approach for hosting official MCP SDK-based servers. This approach is about hosting MCP in Connector Namespace, which was announced in public preview at Build.


Option 3: Queue-based Azure Functions tools

This is the option that gets overlooked because it does not follow the MCP pattern at all, and that is sometimes exactly what you want.

Instead of an AI agent calling your function via the MCP protocol, the agent sends a message to a queue (Service Bus or Azure Storage Queue), and a queue-triggered function picks it up asynchronously. Foundry provides Azure Functions-specific tooling for this pattern.

The Microsoft Learn documentation lists the scenarios where this is the right choice:

  • Reliable message delivery and processing
  • Decoupling between AI agents and function execution
  • Built-in retry and error handling
  • Integration with existing Azure messaging infrastructure

That last point is often what decides it in enterprise contexts. If your integration platform already uses Service Bus for order processing, claims workflows, or event-driven pipelines, the queue-based tool pattern slots into that infrastructure without adding a new protocol layer. Your AI agent becomes another message producer; your function is another consumer. Operations teams already know how to monitor and manage it.

The decoupling is also genuinely useful for long-running tool operations. An MCP call is synchronous from the agent’s perspective: it sends a request and waits for a response. A queue-based call lets the agent fire and move on; the function processes in the background and the result arrives via a separate callback or polling mechanism. For operations that take seconds or minutes document processing, batch retrieval, report generation that asymmetry matters.

Use this when: the tool operation is long-running, you need reliable delivery with built-in retry, you are integrating with existing Azure messaging infrastructure, or you want to decouple the agent from function execution timing.

Avoid it when: the agent needs a synchronous response immediately, or you are connecting to AI clients that expect the MCP protocol specifically.

Note: For long-running processes or multi-step workflows, you might also want to consider Durable Functions instead of “vanilla” Functions. Durable Functions provides built-in state persistence and automatic retries, which is helpful when long-running or multi-step workflows fail mid-execution because of things like unreliable network connectivity. Because state is persisted during execution, Durable Functions can rebuild local state up to the point of failure and continue executing from there instead of from scratch. 


How to choose

SituationOption
Building a new MCP server from scratchBinding extension
Need stateful tool executionBinding extension
Have an existing SDK-based MCP server (Streamable HTTP, stateless)Self-hosted SDK
Need async, decoupled, fault-tolerant tool executionQueue-based
Integrating with existing Service Bus infrastructureQueue-based
Building for GitHub Copilot or VS Code agent modeBinding extension
Production deadline in the next quarterBinding extension or queue-based (avoid preview)
I need pre-warmed instances and predictable latencyElastic Premium plan

The preview constraint is the most important practical point here. If you are building for production and your timeline does not allow for configuration changes mid-project, self-hosted MCP servers are not ready. The binding extension and queue-based options are both GA and stable.

APIM as the governance layer

Whichever option you choose, consider putting APIM in front of your MCP server endpoint. This gives you rate limiting, authentication policy, token quota management, and a single point for logging and monitoring across all MCP tool calls.

One practical caveat from real deployment experience: the Azure AI Foundry Agent Service SDK routes LLM calls directly to Azure OpenAI, bypassing APIM entirely. Tool calls to your MCP server do pass through APIM, but model calls do not unless you use the standard OpenAI SDK instead. The Citadel Platform series covers this in detail, including how to structure your APIM policies to handle the tool-call traffic that does flow through it.

What comes next

The next post goes deeper into the serverless agents runtime, the preview programming model that lets you define event-triggered agents as function apps, with .agent.md files, agents.config.yaml, and remote MCP server connections declared in mcp.json.

Up next: The Azure Functions Serverless Agents Runtime: What It Is and When to Use It

Azure Functions AI Integration: The Quiet Powerhouse

When people talk about running AI workloads on Azure, the conversation usually lands on Azure AI Foundry, Azure OpenAI Service, or maybe Azure Container Apps. Azure Functions tends to get mentioned as the glue the thing you bolt on to handle a webhook. But this overlooks the real story: Azure Functions AI integration has quietly evolved from simple glue into a powerhouse for running production-grade AI.

That framing is outdated. Azure Functions is now a first-class runtime for AI workloads, with four distinct patterns that the Microsoft Learn documentation lays out explicitly. This post walks through each of them and helps you decide which one fits your situation.

The four AI-enabled scenarios

Microsoft groups Azure Functions AI integration into four scenarios:

  1. Serverless agents runtime — event-driven agents that run on serverless infrastructure
  2. Tools and MCP servers — hosting remote Model Context Protocol servers and AI tools
  3. Agentic workflows — multistep, long-running directed agent operations via Durable Functions
  4. Retrieval-augmented generation (RAG) — fast, parallel data retrieval for knowledge-augmented AI

These are not just marketing buckets. Each one reflects a different architectural decision. Let me unpack them.

Azure Functions AI integration: Serverless agents runtime

The serverless agents runtime is a preview programming model for building event-driven agents as function apps. Moreover, Agents are defined in .agent.md files, app-wide runtime defaults live in agents.config.yaml, and remote MCP server connections are listed in mcp.json. The runtime discovers these files, registers the required triggers and endpoints, and runs the agent through the Microsoft Agent Framework when an event fires.

That is a meaningfully different model from what you get in Azure AI Foundry Agent Service. In Foundry, the managed service hosts and orchestrates your agents. In addition, in the serverless agents runtime, your function app is the agent host running on Flex Consumption, with built-in managed identity, monitoring, and scale-to-zero. Furthermore, you write custom Python tools for app-specific logic, and the platform wires in MCP-enabled connections based on Azure connectors and remote MCP servers.

Use this when: you want agents triggered by events, schedules, messages, or HTTP requests and you need the familiar Functions deployment and hosting model rather than a managed agent service.

Avoid it when: you need a fully managed, enterprise-grade agent service with built-in tooling and long-term Microsoft support guarantees today. That is what Foundry Agent Service is built for.

Azure Functions AI integration: Tools and MCP servers

The Model Context Protocol (MCP) has become the industry standard for how AI models and agents interact with external systems. Azure Functions has first-class support for hosting remote MCP servers, and this is already generally available.

There are two hosting options:

OptionStatusHow it works
MCP binding extensionGAUses Functions triggers and bindings; supports stateful execution
Self-hosted MCP servers (MCP SDK)PreviewUses standard MCP SDKs via custom handlers; requires Streamable HTTP transport

The binding extension is the right default. It supports C#, Python, TypeScript, JavaScript, and Java, and it integrates with the Functions programming model you already know. Self-hosted MCP servers offer portability: you can use the official MCP SDKs and bring in existing server code. However, stateful execution is not yet supported, and the configuration is still changing during preview.

There is also a third option worth knowing about: queue-based Azure Functions tools, where AI agents interact with your code through message queues rather than direct MCP calls. Microsoft Foundry provides specific Azure Functions tooling for this pattern. It is ideal when you need reliable delivery, built-in retry, and decoupling between agent and function execution.

Use MCP servers when: you are exposing tools to AI clients and you want the industry-standard protocol with serverless hosting.

Use queue-based tools when: you need asynchronous, fault-tolerant communication between an agent and your function code.

Agentic workflows with Durable Functions

Not all AI orchestration should be autonomous. Some scenarios need predictable, directed steps and that is where Durable Functions fits.

The Microsoft Learn documentation makes the distinction clearly: Durable Functions is positioned as the runtime for directed agentic workflows, not for emergent agent reasoning. Think of it this way: when you know the sequence of steps and you need fault tolerance, auditability, and long-running execution, Durable Functions is the right tool. When you want a model to figure out the steps dynamically, you want an agent runtime.

The documentation gives a clean example: a trip planning workflow that gathers user requirements, searches for options, waits for approval, and makes bookings. Each step is a function; Durable Functions coordinates them with built-in retry, state persistence, and human-in-the-loop support.

Use this when: your AI-driven process has well-defined, ordered steps,authorization flows, multi-stage approval chains, or orchestrated data pipelines where you cannot afford unpredictable execution paths.

Avoid it when: you want a model to determine steps dynamically. That is the serverless agents runtime or Foundry Agent Service territory.

RAG with Azure Functions

Because Functions handles multiple events from various data sources simultaneously, it scales well for real-time AI scenarios, particularly RAG systems where fast, parallel retrieval is the bottleneck.

The Azure OpenAI binding extension lets you integrate RAG directly into your function code. Functions can pull data from multiple sources simultaneously, feed it through Azure AI Search or other retrieval layers, and pass the results to your language model, all within the event-driven, scale-to-zero model that keeps costs down when load is low.

The Azure Functions RAG pattern also pairs naturally with APIM, which handles routing, rate limiting, and token quota management a pattern the Citadel Platform series covers in detail, including the discovery that the Foundry Agent Service SDK bypasses APIM for LLM calls.

Use this when you have event-driven retrieval requirements new documents arriving in blob storage, database change feeds, or streaming IoT data that needs to inform model responses.

How the scenarios relate to other Azure services

It helps to think of Azure Functions AI integration as filling the compute and integration layer between your AI services and your data sources. Here is roughly how that maps:

  • Azure AI Foundry Agent Service — fully managed agent orchestration with enterprise security and built-in tools. Functions integrates into Foundry via MCP servers and queue-based tools.
  • Azure Logic Apps — low-code orchestration for business process automation. Functions is the right choice when you need custom code, complex event processing, or lower latency.
  • Azure Container Apps — container-based hosting for long-running services. Functions on Flex Consumption beats it on cost for bursty, event-driven AI workloads that spend time idle.
  • Durable Functions — lives inside Functions and adds stateful, long-running orchestration. Use it for directed agentic workflows; use the serverless agents runtime for event-driven agents.

The underlying platform advantage

Across all four scenarios, the same hosting model applies: Flex Consumption. It offers fast, event-driven scaling, virtual network integration, and pay-as-you-go billing. For AI workloads, which tend to be bursty rather than continuous, this is a significant cost advantage over always-on hosting.

Managed identity, Application Insights integration, and azd-based deployment are consistent across all four patterns. That means your security posture, observability, and deployment pipeline do not have to change when you move from a simple timer trigger to hosting a remote MCP server.

Azure Functions AI integration: Choosing the right pattern

Here is a simple decision table:

I want to…Use…
Build event- or schedule-triggered agents with MCP toolsServerless agents runtime (preview)
Expose tools to AI clients via the industry-standard protocolMCP binding extension (GA)
Orchestrate predictable, multistep AI-driven processesDurable Functions
Build a RAG pipeline with fast, parallel data retrievalAzure Functions + Azure OpenAI binding
Fully managed agent hosting with enterprise SLAsAzure AI Foundry Agent Service

What comes next

The rest of this series goes deep on each pattern. The next post covers the two MCP server hosting options in detail binding extension versus self-hosted SDK servers, including where the current preview constraints matter in practice.

Up next: Hosting Remote MCP Servers in Azure Functions: GA vs. Preview Options

Building Azure Logic Apps Agent Tools: Connectors and MCP

Part 4 of 7 in the Logic Apps Agent Loop series

Part 3 covered the two agentic workflow patterns in Azure Logic Apps, autonomous and conversational, and how to choose between them. Both patterns rely on the same mechanism for getting work done: tools. An Azure Logic Apps agent loop tool is the means by which the model reaches out to the world to query a database, send an email, call an API, or retrieve a document. Without tools, the agent can only reason over what the model already knows.

This post is the most hands-on in the series. It covers the three layers of the Azure Logic Apps tooling model, built-in connectors, custom connectors, and MCP servers. Moreover, it includes a demo showing how to expose a Logic Apps workflow as a tool provider that can be called by an external agent in Azure AI Foundry.

Choosing the right Azure Logic Apps agent tools layer

Before building anything, it is important first to understand what a tool actually is in Logic Apps terms. Specifically, a tool is defined as a sequence of one or more connector actions that the agent can choose to invoke during a loop iteration. Consequently, the model decides which tool to call based on the tool’s name and description. Therefore, naming and describing tools clearly is one of the most crucial decisions you will make when building an agentic workflow.

Logic Apps offers three layers of tooling, each adding capability and complexity.

Layer 1: Built-in and managed connectors

The foundation layer is the 1,400+ connector library that Logic Apps has always offered. For agent tools, the most relevant connectors are those that give the agent access to data and services: Azure OpenAI, Azure AI Search, Azure Blob Storage, Office 365 Outlook, SharePoint, SQL Server, HTTP, and Service Bus among them.

You build a tool by adding one or more of these connector actions inside the tool container within the agent action. Each tool gets a name and a description. The model reads these at runtime to decide whether to invoke the tool and what arguments to pass. You then create agent parameters for any action inputs that the model should supply dynamically: a city name for a weather lookup, a query string for a search, a recipient address for an email.

Agent parameters differ from standard Logic Apps parameters importantly. They are scoped to the tool where you define them; they cannot be shared across tools. They also receive their values only when the agent invokes the tool, not at workflow start time. You can call the same tool multiple times in a single loop using different parameter values: for example, you could invoke a weather tool for both Amsterdam and London in the same run.

Layer 2: Custom connectors

Where the built-in connector library has gaps, custom connectors fill them. A custom connector in Logic Apps is an OpenAPI-described wrapper around any REST API, internal or external. Furthermore, once you register it, it appears in the connector gallery just like a managed connector, and you can use it inside a tool in the same way.

For enterprise integration architects, custom connectors are the bridge between the agent loop and any internal system that does not have a first-party Logic Apps connector: an internal HR system, a legacy claims processing API, a proprietary data platform. The investment in defining the OpenAPI specification pays off because the connector becomes reusable across all workflows in the tenant, not just the agentic ones.

Building a custom connector for use in an agent tool follows the standard Logic Apps custom connector creation process:: define the API, specify authentication, and configure the operations, with one addition: write clear operation descriptions, because the model uses these descriptions to decide when to invoke the connector.

Layer 3: MCP servers

The third layer is the newest and the most architecturally significant. Azure Logic Apps can serve as the backend for a Model Context Protocol (MCP) server exposing connector actions as a structured, discoverable toolset that external agents and models can call over a standard protocol.

MCP is an open standard that defines how AI components discover and invoke tools. Moreover, an MCP server acts as a bridge between an AI agent and the tools it can use. This is a significant shift from the previous two layers. Built-in and custom connectors are tools that the agent in your Logic Apps workflow invokes. An MCP server inverts the relationship: your Logic Apps workflow becomes the tool provider, and the calling agent lives somewhere else entirely.

A note on the demo: real-world limitations of the tooling preview

For this post I set out to build a working end-to-end demo showing a Logic Apps workflow exposed as an MCP tool provider callable by an Azure AI Foundry agent. The concept is sound and the architecture is correct, but two practical blockers prevented a clean demo at the time of writing.

API Center MCP wizard limitations. The registration wizard in Azure API Center is in active preview. The connector picker surfaces only managed connectors, so the built-in HTTP action from Part 2 is unavailable. The logic app dropdown is also filtered by region, a logic app in West Europe will not appear in an API Center resource deployed to a different region.

Foundry OpenAPI tool network restrictions. Azure AI Foundry’s OpenAPI tool sandbox cannot reach azurewebsites.net endpoints directly. Calls from the Foundry playground return an Unknown error regardless of the spec configuration. The workaround is to front the Logic Apps endpoint with Azure API Management, which Foundry can reach however that adds infrastructure complexity beyond the scope of this post.

Both limitations are preview-stage issues that Microsoft will likely resolve. The OpenAPI spec, the Foundry agent configuration, and the mcp-research workflow pattern described above are all correct and will work once network access between Foundry and Logic Apps endpoints is available or via an APIM gateway.

The Layer 3 pattern of your Logic App as a tool provider for any external MCP-compatible agent remains the most architecturally significant development in this series. In addition, Part 6 picks up the security implications of that expanded caller surface.

Choosing the right tooling layer

The table below summarises how Azure Logic Apps agentic workflows differ across the three tooling layers.

Built-in connectorsCustom connectorsMCP server
Who calls the toolAgent in your workflowAgent in your workflowAny external MCP-compatible agent
Setup complexityLowMediumMedium–high
ReusabilityWithin the workflowAcross the tenantAcross agents and platforms
Best forStandard integrationsInternal APIs without a connectorMulti-agent, cross-platform tooling

The three layers are not mutually exclusive. A production agentic workflow will typically use built-in connectors for standard integrations, custom connectors for internal systems, and an MCP server where the toolset needs to be shared across multiple agents or platforms.


What comes next

The next post moves from individual tools to multi-agent composition. Part 5 covers orchestrator-worker topologies, agent handoffs, and how to build sequential agent loops.

Azure API Management as MCP Gateway: Governing Agentic AI Workloads

Part 7 of 7 in the “APIM for AI Workloads” series

Azure API Management as MCP gateway is the natural endpoint of everything this series has built. In Parts 1 through 6, we established APIM as the control plane for AI workloads: securing access, limiting and measuring token consumption, routing traffic resiliently across backends, and reducing costs through semantic caching. All of that applies equally to agentic workloads. The difference is that agents introduce a new communication pattern: the Model Context Protocol (MCP), which standardizes how AI agents discover and call tools.

In my work and online research on agentic AI architecture, I consistently returned to the same question: how does one govern agent tool calls with the same rigor we apply to API calls? The answer, increasingly, is that APIM handles both. This post covers what that looks like in practice.

What MCP Is and Why It Changes the APIM Story

MCP is an open protocol, originally developed by Anthropic, that defines a standard interface between AI agents (MCP clients) and the tools they call (MCP servers). Instead of each agent framework implementing its own bespoke tool-calling mechanism, MCP gives agents a consistent way to discover available tools, understand their input schemas, and invoke them. Frameworks including Semantic Kernel, AutoGen, and LangGraph are all adding MCP client support.

For APIM, MCP matters because it transforms the gateway from a proxy for AI completions into a broker for agent tool calls. An agent no longer calls your internal APIs directly. Instead, it discovers them as MCP tools through APIM, and APIM enforces the same governance policies on those tool calls that it enforces on any other request. The control plane extends naturally into the agentic layer.

Azure API Management as MCP Gateway: Three Capabilities

APIM’s MCP gateway capabilities fall into three categories:

Expose REST APIs as MCP servers. The export-rest-mcp-server policy takes any API already registered in your APIM catalog and auto-generates MCP tool definitions from it. An agent connecting to your APIM MCP endpoint discovers those tools via the standard MCP protocol and can call them without any knowledge of the underlying REST implementation. Crucially, no changes are required to the underlying API. The policy handles the translation layer entirely within APIM.

Pass through external MCP servers. APIM can proxy external MCP servers — whether third-party services like GitHub or Jira, or custom MCP servers built by your own teams — through the same gateway. All traffic passes through APIM’s policy pipeline, so you apply JWT validation, subscription key enforcement, token limits, and logging to external MCP calls exactly as you would to any other API call. Agents get a single APIM endpoint; APIM handles the routing.

Agent-to-agent (A2A) traffic. In multi-agent architectures, orchestrator agents call sub-agents to delegate tasks. Routing that traffic through APIM means every A2A hop is governed: authenticated, rate-limited, logged, and subject to the same token budget controls applied to end-user traffic. This is particularly relevant for agentic pipelines running on Microsoft Foundry, where multiple specialized agents collaborate within a single workflow.

Applying Series Policies to Agentic Workloads

One of the practical advantages of routing MCP traffic through APIM is that every policy covered in this series applies without modification. Agentic workloads are not a special case requiring a separate governance layer. They use the same pipeline.

  • Authentication (Part 2): Agents authenticate to APIM using subscription keys or JWT tokens. APIM authenticates to AI backends via Managed Identity. The agent never holds backend credentials.
  • Token limits (Part 3): Multi-step agentic pipelines can consume large token volumes per workflow. Per-subscription TPM limits prevent a single runaway pipeline from exhausting shared capacity.
  • Token metrics (Part 4): Token consumption from agentic workflows is attributed to the subscribing team or pipeline via the emit-token-metric policy. FinOps visibility extends automatically to agentic workloads.
  • Load balancing (Part 5): Agentic pipelines often run longer and consume more tokens per call than chat applications. PTU-to-PAYG failover protects pipeline continuity when primary capacity saturates.
  • Semantic caching (Part 6): Agents that make repeated identical tool calls, checking a status, or looking up a reference value, benefit from semantic caching in the same way chat applications do.

Practical Considerations for APIM as MCP Gateway

A few agentic-specific considerations are worth calling out before you start routing MCP traffic through APIM.

Tool discovery latency. MCP clients typically discover available tools at session start by calling the MCP server’s tool list endpoint. With APIM in the path, that discovery call passes through the full policy pipeline. Keep your inbound policies lightweight for discovery calls, or cache the tool list response to avoid repeated round trips.

Streaming responses. Many AI completions endpoints support streaming via server-sent events. APIM supports streaming passthrough, but some policies — including semantic cache lookup — do not apply to streaming responses. Structure your pipeline accordingly: apply caching only to non-streaming completion calls.

Session state. MCP conversations are stateful within a session. APIM is stateless between requests, so per-session state must live in the calling agent or an external store. The vary-by pattern from the semantic cache policy can scope cached tool responses by session ID if the agent passes one in a header.

Token budget propagation. In multi-agent pipelines, token budgets need to propagate from the orchestrator to sub-agents. Exposing the remaining token budget from the remaining-tokens-variable-name attribute (Part 3) as a response header lets orchestration frameworks like Semantic Kernel make informed decisions about which sub-agent to invoke next.

Azure API Management as MCP Gateway: Closing the Series

This post closes the series, but the control plane it describes is not static. MCP is still evolving rapidly. New APIM policy capabilities for agentic workloads are shipping frequently. The architecture board conversation at various enterprise has shifted from “should we centralize AI traffic through APIM?” to “what do we govern next?”, which is a good place to be.

Looking back across the seven posts, the consistent theme is that AI workloads are not fundamentally different from other API workloads in terms of governance requirements. They need authentication, rate limiting, observability, resilience, and cost control. APIM provides all of those. What changes with AI is the unit of measurement (tokens, not requests), the billing model (PTU vs. PAYG), and now the communication protocol (MCP for agents). The control plane adapts to each of these without requiring a parallel governance infrastructure.

The full series index is below for reference. Each post links to the relevant Microsoft documentation and includes policy XML you can use directly.

  • Part 1: Why your AI APIs need a gateway.
  • Part 2: Authentication and authorization.
  • Part 3: Token limit policy.
  • Part 4: Token metric policy and cross-charging.
  • Part 5: Load balancing and circuit breaking.
  • Part 6: Semantic caching.

Part 7 (this post): APIM as MCP gateway for agentic AI workloads.