Azure API Management Build 2026 AI Gateway: What’s New

Posted on June 18, 2026 by steefjan1970

The Azure API Management Build 2026 AI gateway announcements mark a significant expansion of APIM’s control plane capabilities. Microsoft shipped three headline additions: a Unified Model API that lets clients standardize on one format while APIM transforms requests to Anthropic, Google Vertex AI, and other backends; content safety policies extended to cover MCP tool calls and agent-to-agent traffic; and expanded token metrics that now track reasoning, cached, and audio tokens across providers. This post explains what each change means in practice for teams building enterprise AI workloads on Azure.

Azure API Management Build 2026 AI Gateway: Three Headline Changes

The biggest announcement is the Unified Model API, now in public preview. It lets clients standardize on a single API format, currently OpenAI Chat Completions. At the same time, APIM transparently converts requests to the backend provider’s native format, whether that is Anthropic’s Messages API, Google Vertex AI, or another provider.

For teams running multi-model architectures, this is significant. Until now, switching providers or adding a new model required client-side changes. With the Unified Model API, the routing decision moves entirely to APIM. Teams can swap backends, add providers, or route traffic based on cost or latency without touching client code.

From an architecture perspective, this strengthens the case for APIM as the single AI control plane. Every governance policy, rate limit, content safety, and token metric applies consistently regardless of which provider handles inference. There is no need for a parallel governance stack per provider.

One practical implication: the three-layer auth model from Part 2 of this series applies uniformly across all providers. Managed Identity to backend is the cleanest approach, but the provider must support it. For Anthropic and Vertex AI, check the current authentication requirements before assuming token-based auth transfers directly.

Content Safety for MCP and A2A: The Gap That Needed Closing

Extending the llm-content-safety policy to MCP tool calls and agent-to-agent payloads is the most architecturally significant change. Until now, content safety only covered LLM completions traffic. MCP tool-call arguments and A2A messages were ungoverned at the gateway layer.

This matters because prompt injection attacks do not only arrive via the user-facing chat interface. A malicious payload embedded in a tool response from an external MCP server, for example, can propagate through an agentic pipeline if there is no inspection at the gateway layer. The shield-prompt attribute specifically addresses this by checking for adversarial prompt-injection patterns in MCP and A2A traffic, not just in LLM input.

One implementation detail worth calling out: the policy behaves differently for streaming responses. In non-streaming mode, a violation returns a clean 403. In streaming mode, the policy buffers events in a sliding window and stops forwarding without returning an explicit error code. Agents consuming streaming completions need to handle an abrupt stop gracefully. If you are designing agentic pipelines that use streaming, build in a timeout and an explicit error handling path for this case.

The two new attributes — window-size and window-overlap-size — let you tune how content exceeding Azure Content Safety’s 10,000 character limit is split for evaluation. For agentic pipelines with large tool responses, these will need tuning based on your typical payload sizes.

Expanded Token Metrics: Catching What Was Missing

The token metric policy from Part 4 of this series now logs reasoning tokens, cached tokens, and audio tokens to Application Insights. This is a meaningful improvement for FinOps visibility.

Reasoning models like o1 and o3 consume significant token budgets in their internal reasoning chain before producing output. Without reasoning token tracking, cross-charging dashboards systematically undercount consumption from teams using these models. The expanded metrics fix this.

Cached token tracking is equally important for cost optimization. Azure OpenAI’s prompt caching reduces the cost of repeated prompt prefixes. Tracking cached vs. uncached tokens separately lets you measure the actual cache hit rate and tune your prompt structure accordingly.

The multi-provider coverage of Microsoft Foundry, OpenAI, Amazon Bedrock, and Google Vertex AI means the FinOps dashboard built in Part 4 now works across your entire model estate, not just Azure OpenAI.

API Center MCP Server: Enterprise Discovery at GA

The Azure API Center data plane MCP server reached general availability. It acts as a unified discovery endpoint: agents and developer tools can find registered MCP servers, tools, APIs, and AI assets through a single MCP connection. When a team registers a new MCP server in API Center, it becomes automatically discoverable without requiring individual client reconfigurations.

This is the enterprise catalogue layer that makes the MCP gateway story from Part 7 operationally sustainable at scale. Without it, discovery is a manual configuration problem. With it, the control plane extends automatically as new capabilities are registered.

Where This Leaves the Control Plane

Looking at the Build announcements together, the pattern is consistent with what the series argued: APIM is becoming the governance layer for all AI traffic, not just LLM completions. The Unified Model API extends it across providers. Content safety for MCP and A2A extends it across protocols. The API Center MCP server extends discovery to the enterprise catalogue layer.

The competitive context is worth noting. AWS Bedrock Guardrails handles content filtering but has no equivalent to the Unified Model API or MCP/A2A coverage. Google Apigee has added AI gateway features, but not at this protocol breadth. Cloudflare’s AI Gateway focuses on spend limits and caching. APIM’s position that the API gateway is the natural control plane for AI workloadsis increasingly defensible.

For teams that have followed the series and implemented the seven patterns, the Build announcements are additive rather than disruptive. The policy pipeline you built still works. The new capabilities slot in: swap your backend URL configuration to use the Unified Model API, add the llm-content-safety policy to your MCP server inbound pipeline, and update your Application Insights queries to include reasoning and cached token dimensions.

Lastly, the Microsoft AI Gateway labs‘ 30+ Jupyter notebooks with deployable Bicep templates are worth bookmarking if you are implementing any of these patterns.

Azure API Management as MCP Gateway: Governing Agentic AI Workloads

Posted on June 5, 2026 by steefjan1970

Part 7 of 7 in the “APIM for AI Workloads” series

Azure API Management as MCP gateway is the natural endpoint of everything this series has built. In Parts 1 through 6, we established APIM as the control plane for AI workloads: securing access, limiting and measuring token consumption, routing traffic resiliently across backends, and reducing costs through semantic caching. All of that applies equally to agentic workloads. The difference is that agents introduce a new communication pattern: the Model Context Protocol (MCP), which standardizes how AI agents discover and call tools.

In my work and online research on agentic AI architecture, I consistently returned to the same question: how does one govern agent tool calls with the same rigor we apply to API calls? The answer, increasingly, is that APIM handles both. This post covers what that looks like in practice.

What MCP Is and Why It Changes the APIM Story

MCP is an open protocol, originally developed by Anthropic, that defines a standard interface between AI agents (MCP clients) and the tools they call (MCP servers). Instead of each agent framework implementing its own bespoke tool-calling mechanism, MCP gives agents a consistent way to discover available tools, understand their input schemas, and invoke them. Frameworks including Semantic Kernel, AutoGen, and LangGraph are all adding MCP client support.

For APIM, MCP matters because it transforms the gateway from a proxy for AI completions into a broker for agent tool calls. An agent no longer calls your internal APIs directly. Instead, it discovers them as MCP tools through APIM, and APIM enforces the same governance policies on those tool calls that it enforces on any other request. The control plane extends naturally into the agentic layer.

Azure API Management as MCP Gateway: Three Capabilities

APIM’s MCP gateway capabilities fall into three categories:

Expose REST APIs as MCP servers. The export-rest-mcp-server policy takes any API already registered in your APIM catalog and auto-generates MCP tool definitions from it. An agent connecting to your APIM MCP endpoint discovers those tools via the standard MCP protocol and can call them without any knowledge of the underlying REST implementation. Crucially, no changes are required to the underlying API. The policy handles the translation layer entirely within APIM.

Pass through external MCP servers. APIM can proxy external MCP servers — whether third-party services like GitHub or Jira, or custom MCP servers built by your own teams — through the same gateway. All traffic passes through APIM’s policy pipeline, so you apply JWT validation, subscription key enforcement, token limits, and logging to external MCP calls exactly as you would to any other API call. Agents get a single APIM endpoint; APIM handles the routing.

Agent-to-agent (A2A) traffic. In multi-agent architectures, orchestrator agents call sub-agents to delegate tasks. Routing that traffic through APIM means every A2A hop is governed: authenticated, rate-limited, logged, and subject to the same token budget controls applied to end-user traffic. This is particularly relevant for agentic pipelines running on Microsoft Foundry, where multiple specialized agents collaborate within a single workflow.

Applying Series Policies to Agentic Workloads

One of the practical advantages of routing MCP traffic through APIM is that every policy covered in this series applies without modification. Agentic workloads are not a special case requiring a separate governance layer. They use the same pipeline.

Authentication (Part 2): Agents authenticate to APIM using subscription keys or JWT tokens. APIM authenticates to AI backends via Managed Identity. The agent never holds backend credentials.
Token limits (Part 3): Multi-step agentic pipelines can consume large token volumes per workflow. Per-subscription TPM limits prevent a single runaway pipeline from exhausting shared capacity.
Token metrics (Part 4): Token consumption from agentic workflows is attributed to the subscribing team or pipeline via the emit-token-metric policy. FinOps visibility extends automatically to agentic workloads.
Load balancing (Part 5): Agentic pipelines often run longer and consume more tokens per call than chat applications. PTU-to-PAYG failover protects pipeline continuity when primary capacity saturates.
Semantic caching (Part 6): Agents that make repeated identical tool calls, checking a status, or looking up a reference value, benefit from semantic caching in the same way chat applications do.

Practical Considerations for APIM as MCP Gateway

A few agentic-specific considerations are worth calling out before you start routing MCP traffic through APIM.

Tool discovery latency. MCP clients typically discover available tools at session start by calling the MCP server’s tool list endpoint. With APIM in the path, that discovery call passes through the full policy pipeline. Keep your inbound policies lightweight for discovery calls, or cache the tool list response to avoid repeated round trips.

Streaming responses. Many AI completions endpoints support streaming via server-sent events. APIM supports streaming passthrough, but some policies — including semantic cache lookup — do not apply to streaming responses. Structure your pipeline accordingly: apply caching only to non-streaming completion calls.

Session state. MCP conversations are stateful within a session. APIM is stateless between requests, so per-session state must live in the calling agent or an external store. The vary-by pattern from the semantic cache policy can scope cached tool responses by session ID if the agent passes one in a header.

Token budget propagation. In multi-agent pipelines, token budgets need to propagate from the orchestrator to sub-agents. Exposing the remaining token budget from the remaining-tokens-variable-name attribute (Part 3) as a response header lets orchestration frameworks like Semantic Kernel make informed decisions about which sub-agent to invoke next.

Azure API Management as MCP Gateway: Closing the Series

This post closes the series, but the control plane it describes is not static. MCP is still evolving rapidly. New APIM policy capabilities for agentic workloads are shipping frequently. The architecture board conversation at various enterprise has shifted from “should we centralize AI traffic through APIM?” to “what do we govern next?”, which is a good place to be.

Looking back across the seven posts, the consistent theme is that AI workloads are not fundamentally different from other API workloads in terms of governance requirements. They need authentication, rate limiting, observability, resilience, and cost control. APIM provides all of those. What changes with AI is the unit of measurement (tokens, not requests), the billing model (PTU vs. PAYG), and now the communication protocol (MCP for agents). The control plane adapts to each of these without requiring a parallel governance infrastructure.

The full series index is below for reference. Each post links to the relevant Microsoft documentation and includes policy XML you can use directly.

Part 1: Why your AI APIs need a gateway.
Part 2: Authentication and authorization.
Part 3: Token limit policy.
Part 4: Token metric policy and cross-charging.
Part 5: Load balancing and circuit breaking.
Part 6: Semantic caching.

Part 7 (this post): APIM as MCP gateway for agentic AI workloads.

Azure API Management Semantic Caching: Cut AI Token Costs with Similarity-Based Responses

Posted on June 3, 2026 by steefjan1970

Part 6 of 7 in the “APIM for AI Workloads” series

Azure API Management semantic caching is the most operationally transparent cost optimization in this series. Every technique covered so far, auth, token limits, token metrics, and load balancing, requires deliberate design decisions in how you configure APIM. Semantic caching, by contrast, works silently. Calling applications sends prompts as normal. APIM checks whether a semantically similar prompt has already been answered. If a match exists above a configurable similarity threshold, APIM returns the cached response without touching the AI backend. Zero tokens consumed. Zero latency is added by the model.

For workloads with repetitive prompt patterns, internal FAQ bots, document classifiers, and support agents that see the same questions repeatedly, the cache hit rate can be surprisingly high. Even a 20% hit rate on a high-volume workload translates directly into cost reduction and lower average latency.

How Azure API Management Semantic Caching Works

The azure-openai-semantic-cache-lookup policy sits in the inbound section of your APIM pipeline, before the request reaches the AI backend. When a prompt arrives, APIM sends it to a configured embedding model, typically Azure OpenAI text-embedding-ada-002 or equivalent, to generate a vector representation of the prompt. APIM then compares that vector against cached embeddings stored in Azure Managed Redis using cosine similarity.

If the similarity score between the incoming prompt and a cached prompt falls below the configured score-threshold, APIM treats it as a cache hit and returns the stored response. If no match meets the threshold, APIM forwards the request to the AI backend as normal and stores the response in Redis for future lookups.

The generic variant, llm-semantic-cache-lookup, works identically for non-Azure backends. Both require the same supporting infrastructure: an embedding model backend and an Azure Managed Redis instance configured in APIM. The semantic cache store policy handles writing responses back to the cache in the outbound section.

Tuning the Score Threshold for Azure API Management Semantic Caching

The score-threshold attribute is the most consequential configuration decision in the semantic caching policy. It controls how similar an incoming prompt must be to a cached prompt for APIM to treat it as a hit. The value runs from 0.0 to 1.0, but the practical range is much narrower.

In practice, three zones matter:

0.01 to 0.05 (aggressive). At this range, prompts that are paraphrases of each other — “What is my account balance?” and “Can you show me my current balance?” — reliably produce cache hits. This is the right range for FAQ bots, support agents, and any workload where users ask the same questions in slightly different words. The default of 0.05 sits here and suits most production deployments.

0.05 to 0.20 (conservative). At this range, only prompts that are very close in wording produce hits. Creative workloads, code generation, and document drafting tend to have high prompt variance, so a more conservative threshold avoids serving stale cached responses to genuinely different requests.

Above 0.30 (too strict). At this threshold, almost no prompts match. The cache effectively stops functioning. Avoid this range unless you are deliberately disabling caching for a specific API product while keeping the policy in the pipeline for future use.

Start at 0.05 and monitor cache hit rates in Application Insights. If the hit rate is low for a workload you expect to be repetitive, lower the threshold incrementally. If you start seeing complaints about incorrect or stale responses, raise it.

vary-by Scope: Preventing Cache Pollution

The vary-by element scopes the cache namespace. Without it, all consumers share a single global cache. That maximizes the hit rate but introduces a significant risk: APIM could serve one user’s cached response to a different user. For most enterprise AI workloads, that is unacceptable.

The safest default is to vary by Subscription ID, which gives each API subscriber their own cache namespace. This prevents cross-team cache pollution while still achieving high hit rates within each subscriber’s own prompt patterns. For multi-tenant applications where individual users have distinct contexts, vary by a user identifier extracted from the JWT or a custom header instead.

A global cache with no vary-by is appropriate only for fully public, stateless APIs where responses are identical regardless of who requests them. Internal enterprise AI workloads rarely meet that bar.

Infrastructure Requirements for Azure API Management Semantic Caching

Semantic caching requires two supporting Azure resources beyond APIM itself. First, an Azure Managed Redis instance configured as an external cache in APIM. Redis stores the prompt embeddings and cached responses. The cache TTL is configurable in the store policy, so you control how long responses remain valid before APIM re-queries the backend.

Second, an embeddings model backend registered in APIM. For Azure OpenAI, this is typically a separate deployment of text-embedding-ada-002 or text-embedding-3-small. The embeddings backend is referenced by the embeddings-backend-id attribute. It is separate from your completions backend, so you can apply independent token limits and load balancing to the embeddings traffic.

One practical consideration: the embeddings call itself consumes tokens and adds a small amount of latency on every request, whether or not the cache hits. For workloads with very low prompt repetition, the overhead of generating embeddings for every request may outweigh the savings from occasional cache hits. Measure the hit rate before committing the infrastructure cost.

What’s Next in This Azure API Management for AI Series

Part 7 closes the series by covering APIM’s emerging role as an MCP gateway for agentic AI workloads: how to expose REST APIs as MCP servers, pass through existing MCP servers, and manage agent-to-agent traffic through the same control plane we’ve built across this series.

Part 7: APIM as an MCP gateway for agentic AI workloads.

Azure API Management Load Balancing and Circuit Breaker for AI Backends

Posted on May 27, 2026 by steefjan1970

Part 5 of 7 in the “APIM for AI Workloads” series

Azure API Management load balancing for AI workloads solves a problem that every team hits once they move beyond a single Azure OpenAI deployment: PTU capacity is finite, PAYG is a safety net, and when things go wrong on one backend, the rest of your workload should not notice. In Part 1 of this series, I described PTU vs. PAYG as a routing problem. This post is where we solve it.

The combination of backend pools, priority-based routing, and circuit breaker rules in APIM gives you a resilient AI gateway that handles three distinct failure modes: PTU saturation (too many tokens consumed against reserved capacity), regional outages, and transient backend errors. None of these requires changes to calling applications. APIM absorbs the complexity and presents a single stable endpoint.

Azure API Management Load Balancing: Backend Pools for AI

APIM’s backend pool feature lets you define a named group of AI backends and route to them as a unit. You reference the pool in the set-backend-service policy by its pool ID. When a request arrives, APIM selects a backend from the pool based on priority and weight, tracks health state via the circuit breaker, and retries on the next available member if the selected backend fails.

For AI workloads, the standard pattern uses two tiers. The first tier is your PTU deployment reserved capacity in a primary region, assigned priority 1. The second tier is a PAYG deployment in a secondary region, assigned priority 2. APIM routes all traffic to the PTU backend as long as the PTU backend is healthy. When PTU returns a 429 (capacity exceeded) error or becomes unreachable, the circuit breaker trips, and APIM automatically fails over to the PAYG backend.

Priority determines the preference order: lower numbers are preferred. Weight applies when multiple backends share the same priority, distributing load proportionally between them. A common pattern for multi-region PTU deployments is two PTU backends at priority 1, each with a different weight reflecting their provisioned capacity, and a shared PAYG backend at priority 2 as the common overflow.

Circuit Breaker Configuration for Azure API Management AI Backends

The circuit breaker is what makes the backend pool resilient rather than just load-balanced. Without it, APIM continues routing to a saturated or unavailable backend on every request, each one failing with a 429 or timeout before falling back. The circuit breaker short-circuits that path: after a configurable number of failures within a time window, it marks the backend as OPEN and stops sending traffic to it entirely.

The three circuit breaker states map directly to operational behavior:

CLOSED is the normal state. All requests are routed to the backend. Failures APIM counts failures within the configured interval, and the counter resets at the end of each interval if the number of failures remains below the threshold.

After enough failures to exceed the threshold, the breaker trips to OPEN. In this state, APIM bypasses the backend entirely, and APIM routes to the next available pool member without attempting the failed backend again. The tripDuration timer starts counting down immediately.

Once tripDuration elapses, the breaker enters HALF-OPEN and sends a single probe request to test recovery. A successful response transitions the backend back to CLOSED. A failure resets the timer and keeps the circuit OPEN.

For Azure OpenAI specifically, 429 should always be in your failureCondition alongside 503 and 504. A 429 from a PTU endpoint indicates that the provisioned throughput ceiling has been reached and the backend is temporarily unable to serve requests. That is exactly the condition you want to trip the circuit and fail over to PAYG, rather than returning errors to the caller.

Sizing Circuit Breaker Parameters for AI Workloads

The right circuit breaker parameters depend on your traffic pattern and how quickly you need failover to activate. A few practical guidelines:

threshold: For AI workloads, 3 to 5 failures is a reasonable starting point. PTU endpoints return 429 consistently when saturated, so you don’t need a high threshold to detect the condition. Setting it too high means you absorb too many failed requests before failing over.

interval: 60 seconds works well for most workloads. This is the window over which failures are counted. Shorter intervals are more sensitive to transient errors, while longer ones suit bursty traffic patterns where a few failures in a short window are expected.

tripDuration: 30 seconds is a sensible default. PTU capacity refreshes on a per-minute basis, so a 30-second trip duration gives the backend time to recover before the probe fires. For deployments where PTU saturation is a known recurring pattern, a longer trip duration (60 to 120 seconds) reduces the frequency of failed probes.

Retry Policy and Agentic Workload Considerations

Backend pool failover and circuit breaking handle backend-level failures, but you may also want a retry policy in your APIM inbound pipeline for transient errors that do not warrant a full circuit trip. The retry policy can be scoped to specific status codes and configured with a backoff interval, giving you a two-level resilience model: retry for transient errors, circuit break for sustained failures.

For agentic workloads specifically, failover behavior needs careful thought. A conversational agent mid-session that silently switches from a PTU to a PAYG backend will not notice the change at the model API level. But agentic pipelines with multiple sequential tool calls are more sensitive: a mid-pipeline failover can introduce latency spikes that cause timeouts in orchestration layers such as Azure Logic Apps or Semantic Kernel.

The practical mitigation is to expose the remaining token budget via the token limit policy variable from Part 3 and have the orchestration layer monitor it to proactively slow down before circuit breaking kicks in. Prevention is cheaper than recovery when the workload is stateful.

What’s Next in This Azure API Management for AI Series

Part 6 covers semantic caching: how APIM uses an embeddings model and Azure Managed Redis to serve cached responses for semantically similar prompts, reducing token consumption and latency without any changes to calling applications.

Part 6: Semantic caching — reducing token consumption with similarity-based response reuse.
Part 7: APIM as an MCP gateway for agentic AI workloads.

Azure API Management Token Metric Policy: AI Cost Observability and Cross-Charging

Posted on May 20, 2026 by steefjan1970

Part 4 of 7 in the “APIM for AI Workloads” series

The Azure API Management token metric policy turns AI cost data from a finance problem into an engineering one. In Part 3, we covered enforcement: how to set consumption boundaries per consumer. This post covers the complementary piece: how to measure that consumption. More importantly, it shows how to make it visible to the right people and use it to drive internal cross-charging and FinOps dashboards.

At my current company, one of the first questions the architecture board asked was straightforward: which teams are consuming what, and what does it cost? Without instrumentation at the gateway layer, that question is genuinely unanswerable. The token metric policy is how you answer it.

Azure API Management Token Metric Policy: How It Works

The policy sits in the outbound section of your APIM pipeline. After the AI backend returns a response, APIM reads the token usage fields from the response body. These include prompt tokens, completion tokens, and total tokens. APIM then emits them as custom metrics to Application Insights under a namespace you define.

Crucially, the policy emits metrics after the response arrives. It uses actual token counts from the API response rather than estimates. As a result, the data is accurate rather than approximated. It also means the metric emission adds no latency to the request path: the response is returned to the caller immediately, and the metric is emitted asynchronously.

The generic variant, llm-emit-token-metric, works identically for non-Azure backends. Both policies share the same dimension model, so the configuration patterns below apply regardless of which AI provider sits behind APIM.

Choosing Dimensions for Azure API Management Token Metric Policy

Dimensions are the labels attached to each metric event. They explain how to slice and aggregate token consumption data in Application Insights. Choosing the right dimensions is the most important configuration decision for making the data useful for cross-charging.

The three primary dimension options are:

Subscription ID. The most common choice for internal enterprise deployments. Each APIM subscription maps to a team, product, or cost center, so filtering Application Insights metrics by Subscription ID gives you direct per-team token consumption. This pairs naturally with the subscription key authentication pattern from Part 2 and the per-subscription counter-key from Part 3.

User ID. Sourced from the JWT subject claim or a custom header, User ID enables per-user consumption reporting. This is the right dimension for multi-tenant SaaS applications where individual end users have their own token budgets, or where you need to identify heavy consumers within a shared subscription.

API ID. Identifies which APIM API product generated the consumption. Useful when a single subscription uses multiple AI-backed APIs: one for a conversational agent, one for content generation, and one for document summarization. API ID lets you break down cost by use case rather than just by subscriber.

In practice, combining all three dimensions gives you the most flexibility. A single metric event tagged with Subscription ID, User ID, and API ID can answer questions at every level: how much did the platform spend in total, how much did Team A spend, how much did User X consume, and which AI feature is the most expensive to run.

Querying Token Metrics in Application Insights

Once the policy is emitting metrics, you query them in Application Insights using the custom metrics namespace you configured. The metrics appear under the namespace name you set in the policy (for example, “AzureOpenAI” or “MyLLM”), with separate metric events for prompt tokens and completion tokens.

A practical starting point is a KQL query that aggregates the total number of tokens by Subscription ID over the past 30 days. From there, you can add filters by API ID to isolate specific workloads, or pivot by User ID to identify the highest consumers within a team.

For FinOps dashboards, the most useful view is a stacked time-series chart of total token consumption broken down by subscription, updated daily. This gives finance and engineering a shared view of AI spend trends without exporting data from Azure Monitor to a separate BI tool. Azure Workbooks can host this directly in the Azure portal, making it accessible to non-technical stakeholders.

From Observability to Cross-Charging

Observability is the prerequisite for cross-charging. However, they are not the same thing. Observability tells you what happened. Cross-charging, by contrast, is the organizational process of allocating those costs to the right budget owners.

The token metric policy gives you the raw data. To turn that into a cross-charge, you need two additional steps. First, agree on a price per token with your finance team — usually derived from the Azure cost per 1,000 tokens for your model and region. Second, automate a monthly report that multiplies token consumption by the subscription price.

This does not need to be complex. For example, a Logic App or Azure Function that queries Application Insights on the first of each month works well for most organizations starting out. It aggregates tokens by subscription, multiplies by the agreed rate, and emails a cost summary to each team lead. The Application Insights REST API makes this straightforward to automate.

Finally, the most important advice: have this conversation with finance and product teams before AI consumption scales. Retroactive cross-charging is significantly harder to establish than an upfront model with clear methodology and tooling.

What’s Next in This Azure API Management for AI Series

Part 5 covers load balancing and circuit breaking: how to distribute traffic across PTU and PAYG backends, configure backend pools, and set up circuit breaker rules for automatic failover when a primary endpoint becomes unavailable.

Part 5: Load balancing and circuit breaking across PTU and PAYG backends.
Part 6: Semantic caching — reducing token consumption with similarity-based response reuse.
Part 7: APIM as an MCP gateway for agentic AI workloads.

AI Gateway Commercial vs Open Source: How to Choose the Right Control Plane

Posted on May 15, 2026 by steefjan1970

The AI gateway commercial vs. open-source decision is one that most organizations reach not by planning but by accident. One team has already integrated directly with Azure OpenAI. Another is using LiteLLM to wrap a few models. A third wants to use the enterprise API management platform you already have. Suddenly, you need to make a choice, and the conversation gets complicated fast.

This companion post to my APIM for AI Workloads series takes a step back from the Azure API Management specifics and addresses the question that comes before all of it: which gateway should you be using in the first place? The series covers APIM in depth because it’s the right answer for the Microsoft ecosystem. But it’s not the only answer, and for some organizations it’s not the right one.

Here is how to think through the decision properly.

Why the AI Gateway Commercial vs Open Source Choice Matters More Than You Think

Most API gateway decisions are relatively low-stakes. If you pick the wrong one, you migrate. But the AI gateway decision carries more weight for two reasons.

First, the gateway sits in the critical path of every AI interaction in your organization. Its policy language, authentication model, and observability hooks become embedded in the way your teams build AI-powered applications. Switching later is not impossible, but it is disruptive.

Second, the governance patterns you establish now, how you handle token limits, cross-charging, PII, and compliance logging, are much harder to retrofit than to design in from the start. The Team Rockstars IT AI Gateway whitepaper, published this month, makes this point well: organizations that set up audit logging via an AI gateway from day one build a direct compliance advantage under the EU AI Act. Those who add it later risk complex and costly rework.

So the choice deserves deliberate thought, not a default.

The Commercial Options for AI Gateway

Commercial AI gateways offer a faster path to production and offload operational complexity to the vendor. The main options in the market today are:

Azure API Management is the right choice if you are already in the Microsoft ecosystem. Its AI-specific policy extensions for token limits, token metrics, semantic caching, and load balancing across PTU and PAYG backends are mature and tightly integrated with Azure Monitor and Application Insights. The series covers this in depth from Part 1 onwards.

Kong Konnect is a strong option for organizations that already use Kong for API management and want to extend it into AI. Its plugin ecosystem covers rate limiting, authentication, and observability, with AI-specific plugins growing quickly.

Portkey is purpose-built as an AI gateway with a lightweight footprint and fast time-to-value. It supports a broad range of model providers, has built-in semantic caching and observability, and is a practical option for teams that want AI governance without the overhead of a full enterprise API management platform.

Apigee (Google Cloud) is the natural choice for GCP-centric organizations. Like APIM in the Microsoft world, its AI gateway capabilities are deepening with each release as Google embeds Gemini and Vertex AI integrations.

The common advantages across all commercial options are faster deployment, built-in compliance features, vendor support contracts, and operational burden offloaded to the vendor. The common risks are licensing costs, proprietary policy languages that create switching friction, and dependency on the vendor’s roadmap.

The Open Source Options for AI Gateway

Open-source gateways offer maximum control and no licensing costs, but they require your organization to own what the vendor would otherwise handle.

LiteLLM is the most widely adopted open source AI gateway today. It provides a unified API across more than 100 model providers, with built-in rate limiting, spend tracking, and a proxy server that is straightforward to self-host. The community is active, and the feature velocity is high. The supply chain risk is real, though: a 2025 attack targeting LiteLLM and Trivy demonstrated that even widely used security-adjacent tools can become attack vectors. If you run LiteLLM in production, you own the patching cadence.

Agent Gateway from Anthropic is purpose-built for MCP and agentic traffic. If your primary use case is governing tool calls from AI agents rather than managing completion API traffic, it is worth evaluating alongside the broader options.

One API provides a unified, OpenAI-compatible interface across multiple providers and is widely used by organizations seeking provider-agnostic routing without vendor lock-in.

HelixML focuses on self-hosted deployments with strong data-sovereignty properties, making it relevant for organizations where data-residency requirements rule out SaaS-based gateway options.

AI Gateway Commercial vs Open Source: Five Decision Factors

Five factors consistently determine which direction is right for a given organization:

Time to value. In my experience, commercial gateways can be production-ready in days to weeks. Open source deployments typically take weeks to months to reach production quality, depending on how much custom policy logic you need to build. If you have an urgent compliance or cost control problem to solve, commercial is the pragmatic choice.

Compliance and data residency. For Dutch and European organizations operating under AVG, NIS2, and the EU AI Act, commercial gateways offer contractual guarantees: data processing agreements, certified regions, and SLAs with defined incident response times. Open source can meet the same requirements, but you are responsible for demonstrating compliance yourself rather than relying on a vendor certification.

Internal platform capability. Open source is not free. The licensing cost is zero, but according to the CNCF’s platform engineering maturity model. Organizations without a dedicated platform engineering team that can credibly own the gateway long-term should not choose open source. The operational gap will become visible at the worst possible moment.

Flexibility and lock-in risk. Open source wins on long-term flexibility. Proprietary policy languages in commercial gateways create switching friction that grows over time as you invest in custom policies. If multi-cloud strategy and provider-agnosticism are strategic priorities, design your gateway layer with that in mind from the start, even if you begin with a commercial option, applying the strangler fig pattern to abstract away proprietary dependencies over time.

Supply chain risk. This factor is underweighted in most evaluations. The 2025 supply chain attack targeting LiteLLM and Trivy demonstrated that open source security tooling itself can become an attack vector. Commercial vendors have contractual obligations around vulnerability disclosure and patching. With open source, that obligation falls to your team.

A Decision Framework for AI Gateway Commercial vs Open Source

The flowchart above works through the most decisive questions in order. A few practical observations from applying it:

Regulated industries almost always land in commercial. Healthcare, financial services, and insurance organizations operating under Dutch or European regulation have compliance requirements that are significantly easier to satisfy with contractual vendor guarantees than with self-operated open source tooling. At my company, the AVG and healthcare-specific data processing requirements made APIM the clear choice.

The hybrid pattern is underused. Many organizations run a commercial gateway in production for governed workloads, while developer teams use LiteLLM or a lightweight open source option in lower environments for experimentation. This gives you the compliance and operational properties you need in production while keeping the innovation surface open. It is more work to maintain two gateway patterns, but the tradeoff is often worth it.

Design for replaceability regardless of what you choose. The Team Rockstars whitepaper frames this well: choose your first gateway deliberately, but design for replacement. Use open standards, abstract your policy logic where possible, and avoid deep coupling to proprietary features without open-source equivalents. The gateway landscape is evolving fast enough that what is the right choice today may not be in two years.

Where This Fits in the APIM for AI Workloads Series

The rest of the series goes deep on Azure API Management specifically: the token metric policy, load balancing and circuit breaking, semantic caching, and MCP gateway for agentic workloads. If you have landed on APIM as your gateway of choice or if you are in a Microsoft-centric organization where it is the natural fit, the series covers the production patterns you need.

Part 1: Why your AI APIs need a gateway.
Part 2: Authentication and authorization.
Part 3: Token limit policy.
Part 4: Token metric policy and cross-charging.
Part 5: Load balancing and circuit breaking.
Part 6: Semantic caching.
Part 7: APIM as MCP gateway for agentic AI workloads.

Azure API Management Token Limit Policy: Controlling AI Token Consumption Per Consumer

Posted on May 11, 2026 by steefjan1970

Part 3 of 7 in the “APIM for AI Workloads” series

The Azure API Management token limit policy is one of the most direct cost control levers you have for AI workloads. In Part 1 of this series, I argued that token consumption is invisible without the right instrumentation. The token limit policy is the enforcement side of that equation: once you know how many tokens consumers are using, you set boundaries so that no single consumer can exhaust your model capacity or run up an unexpected bill.

This post covers how the policy works, which counter-key strategy to choose for your workload, how to size your tokens-per-minute (TPM) limits, and the difference between the Azure OpenAI-specific policy and the generic LLM variant for non-Microsoft backends.

Azure API Management Token Limit Policy: How It Works

The azure-openai-token-limit policy sits in the inbound section of your APIM policy pipeline. Before any request reaches the AI backend, APIM checks a sliding window counter keyed to the value you specify. If the caller is within their TPM budget, the request passes through. If they’ve exceeded it, APIM returns a 429 Too Many Requests response with a Retry-After header, and the backend never sees the request.

This is important: the throttling happens at the gateway, not at the Azure OpenAI endpoint. That means you’re not paying for rejected requests, and your model deployment is protected from saturation by a single runaway consumer.

The policy has two variants. The azure-openai-token-limit policy is purpose-built for Azure OpenAI and Microsoft Foundry endpoints, and uses the actual token counts returned in the API response. The llm-token-limit policy is the generic variant for any LLM backend, including Mistral, Cohere, and others. Both share the same attribute model, so the configuration patterns below apply to either.

Choosing a counter-key for Azure API Management Token Limiting

The counter-key attribute is the most important decision in configuring the token limit policy. It determines the scope of the limit: who shares a TPM bucket, and who gets their own.

The three main strategies are:

Per subscription: @(context.Subscription.Id). This is the most common pattern for internal enterprise use. Each API product subscription gets its own TPM counter, which maps cleanly to a team, a product, or a cost center. Combined with the Token Metric policy covered in Part 4, this provides per-subscriber cost visibility and enforcement in a single configuration.

Per IP address: @(context.Request.IpAddress). Better suited to public-facing endpoints or developer portals where you don’t have a subscription model. It’s a blunt instrument — NAT and shared egress can mean multiple users share a counter — but it’s effective for abuse prevention and trial access scenarios.

Per JWT claim or custom header: @(context.Request.Headers.GetValueOrDefault(“x-user-id”,””)). The most flexible option. If your application passes a user identifier in a header or JWT claim, you can scope limits to the individual user. This is the right approach for multi-tenant applications where each end user should have their own token budget, independent of which subscription they’re calling through.

Sizing Your TPM Limits

TPM limits are context-dependent, but a few principles apply across most workloads.

Start by profiling your actual token usage in a staging environment before setting production limits. The remaining-tokens-variable-name attribute exposes the remaining token budget as a policy variable, which you can log via the Token Metric policy to build a usage baseline before enforcing hard limits.

For the estimate-prompt-tokens attribute: set it to false in production. When set to true, APIM estimates prompt tokens before the response is returned, enabling earlier throttling but reducing accuracy. In practice, counting actual tokens from the response is more reliable and avoids throttling requests that would have been within budget.

A common mistake is setting a single global TPM limit too low, which throttles all consumers the moment a batch job runs on any team. The better pattern is tiered limits by API product: a Developer product with a low TPM ceiling, a Standard product for normal workloads, and an Unlimited product for production pipelines that need burst capacity.

Handling 429 Responses in Calling Applications

Any application calling an APIM-fronted AI endpoint needs to handle 429 responses gracefully. APIM returns a Retry-After header indicating how many seconds until the token window resets. Well-behaved clients respect this header and back off rather than retrying immediately.

For agentic workloads with multiple pipeline steps, a 429 response midway through can leave the agent in an inconsistent state. The recommended pattern is to expose the remaining-tokens-variable-name value in a response header so the calling application can monitor its own budget and slow down proactively, rather than waiting for a hard rejection.

The Azure OpenAI token limit policy documentation covers the full attribute reference, including tokens-per-minute, counter-key, estimate-prompt-tokens, and remaining-tokens-variable-name. The llm-token-limit variant has the same interface for non-Azure backends.

What’s Next in This Azure API Management for AI Series

Part 4 covers the Token Metric policy: how to emit token usage data to Application Insights broken down by consumer dimensions, and how to use that data for internal cross-charging and spend dashboards.

Part 4: Token Metric policy — emitting usage data for observability and cross-charging.
Part 5: Load balancing and circuit breaking across PTU and PAYG backends.
Part 6: Semantic caching — reducing token consumption with similarity-based response reuse.
Part 7: APIM as an MCP gateway for agentic AI workloads.

Azure API Management for AI: Securing Your AI APIs with Authentication and Authorization

Posted on May 5, 2026 by steefjan1970

Part 2 of 7 in the “APIM for AI Workloads” series

In Part 1 of this series, I made the case for why Azure API Management for AI workloads is the right control plane for governing AI traffic across an organization. This post gets practical: how do you actually secure access to your AI backends with APIM without creating a credential-management nightmare?

Security is where many AI projects cut corners, and understandably so. When you’re moving fast to prove value with a new model, authentication feels like overhead. But AI endpoints are expensive, and an unsecured Azure OpenAI endpoint is a real risk: anyone with the URL and key can start consuming tokens at your cost. At scale, that’s a significant financial and compliance exposure.

APIM addresses this with a three-layer security model. Let’s walk through each layer.

Azure API Management for AI Security: A Three-Layer Model

The authentication and authorization pattern in APIM is deliberately layered. Each layer answers a different question and operates independently, so a failure at any layer stops the request before it reaches the AI backend.

The three layers are:

Subscription keys to identify and track API consumers.
JWT validation to enforce fine-grained access control based on claims.
Managed Identity to authenticate APIM to Azure OpenAI without storing credentials.

Each layer has a distinct role. Confusing them is a common mistake, so it’s worth being explicit about what each one does and does not do.

Layer 1: Subscription Keys

Subscription keys are APIM’s mechanism for identifying API consumers. When you create an API product in APIM and require a subscription, callers must include their key in the Ocp-Apim-Subscription-Key header. APIM validates the key, maps it to a subscriber, and lets the request proceed.

This is important for AI workloads specifically because subscription keys enable per-consumer token tracking. When you combine subscription key validation with the Token Metric policy we’ll cover in Part 4, you get usage data broken down by subscriber, which is the foundation of any internal cross-charging model.

Subscription keys answer the question: Who is calling? They don’t answer what the caller is allowed to do. For that, you need JWT validation.

Layer 2: JWT Validation and Claims-Based Authorization

The validate-jwt policy is where you enforce what a caller is permitted to do. It validates the JWT token in the Authorization header against your identity provider, and can inspect any claim in the token to make authorization decisions.

For Azure OpenAI specifically, this is where you control which teams or applications can access which model deployments. A team working on an internal chatbot should not be able to call a GPT-4o deployment reserved for a production workload. JWT claims let you enforce that boundary at the gateway layer, with no changes required in the calling application.

A typical policy checks the token signature against your Azure AD tenant’s OpenID Connect configuration, then validates that a required scope or role claim is present:

The failed-validation-httpcode=”401″ attribute ensures unauthenticated callers get a clean rejection before they ever reach the backend. You can also use failed-validation-error-message to return a specific error message, which helps consumers debug auth failures without exposing internal details.

For multi-provider setups where you’re routing to non-Azure backends like Mistral or Cohere, the same JWT policy applies. The claims model is provider-agnostic, which is one of the advantages of centralizing auth in APIM rather than handling it per-backend.

Layer 3: Managed Identity for Backend Authentication

Managed Identity is the most important security improvement you can make when setting up Azure API Management for AI. It replaces the pattern of storing an Azure OpenAI API key in APIM’s named values with a system-assigned or user-assigned Managed Identity that APIM uses to authenticate directly to Azure OpenAI via Azure AD.

The practical difference is significant. With API key authentication, you have a long-lived secret that needs to be stored, rotated, and kept out of source control. With Managed Identity, there is no secret. APIM requests a short-lived token from Azure AD at runtime, and Azure AD issues it based on the APIM instance’s identity. Nothing is stored. Nothing can leak.

The configuration is a single policy element in the inbound section: <authentication-managed-identity resource=”https://cognitiveservices.azure.com”/>. APIM handles the rest, automatically fetching and refreshing the token.

On the Azure OpenAI side, you grant the APIM instance’s Managed Identity the Cognitive Services User role on the Azure OpenAI resource. That’s the minimum required permission. You can scope it further to specific deployments if needed.

For organizations in regulated industries, such as healthcare, financial services, and government, Managed Identity is not optional. It satisfies Zero Trust authentication requirements and produces a full audit trail in Azure Monitor, tied to the APIM instance identity rather than a shared key.

Azure API Management for AI: Putting the Three Layers Together

In a production setup, all three layers run sequentially within the inbound policy pipeline. A request arrives with a subscription key and a JWT. APIM validates the key first (fast, no external call), then validates the JWT against Azure AD, then forwards the request to Azure OpenAI using its Managed Identity token. The AI backend never sees the caller’s JWT, and APIM never stores an API key.

The result is a clean separation of concerns:

The calling application manages its own JWT (issued by Azure AD based on its own identity or the user’s identity).
APIM enforces the authorization policy without the backend needing to know anything about it.
The AI backend trusts only APIM’s Managed Identity, not arbitrary callers.

This is the architecture you want before you go to production with any AI workload that touches sensitive data or incurs meaningful cost.

What’s Next in This Series

Part 3 covers the Token Limit policy: how to enforce tokens-per-minute limits per consumer, configure throttling behavior, and handle the differences between the azure-openai-token-limit and llm-token-limit policy variants.

Part 3: Token Limit policy — enforcing tokens-per-minute limits per consumer.
Part 4: Token Metric policy — emitting usage data for observability and cross-charging.
Part 5: Load balancing and circuit breaking across PTU and PAYG backends.
Part 6: Semantic caching — reducing token consumption with similarity-based response reuse.
Part 7: APIM as an MCP gateway for agentic AI workloads.

Azure API Management for AI: Why Your APIs Need a Gateway

Posted on May 2, 2026 by steefjan1970

Part 1 of 7 in the “APIM for AI Workloads” series

Over the past year, I’ve been doing a lot of work with integration services, including Azure API Management and, recently, also on AI adoption: evaluating models, designing agentic architectures, and figuring out how to govern AI consumption across the organization responsibly. One thing that keeps coming up in those conversations is a question that sounds almost too basic to ask: Who is keeping track of what we’re spending on tokens?

The answer, more often than not, is nobody.

That’s the problem this series is about. AI APIs are fundamentally different from the REST APIs we’ve been managing for the past decade, and the differences matter operationally. Before we dive into the mechanics of Azure API Management policies, load balancing, and semantic caching in subsequent posts, I want to make the case for a gateway layer in front of your AI services. Before we dive into the mechanics of Azure API Management for AI workloads, policies, load balancing, and semantic caching, I want to make the case for why you need a gateway layer in front of your AI services.

Tokens Are Not Requests

Traditional API management was built around a relatively simple model: count the requests, enforce rate limits, log the traffic, and call it done. One call in, one response out. The cost model was predictable.

AI APIs broke that model completely.

When you call an Azure OpenAI endpoint, you’re not paying per request. You’re paying per token. And a token count is invisible at the API gateway layer unless you specifically instrument for it. A single call from a conversational agent might consume 500 tokens. A call from a poorly-optimized batch process might consume 50,000. Both look the same at the HTTP level: one POST, one 200 OK.

This creates a blind spot that grows dangerously as AI adoption scales across an organization. Teams start building intelligent apps: conversational agents, personalized content generators, voice assistants, copilots, and each one is independently calling AI backend services. Nobody has a view across the whole estate of what’s being consumed, by whom, and at what cost.

The diagram below shows what this looks like in practice: multiple application types hitting multiple AI providers, with token-based pricing models sitting underneath.

The Three Problems Azure API Management for AI Solves

Azure API Management acts as the centralized control plane between your intelligent applications and your AI backends. It addresses three distinct categories of problems.

Performance optimization: AI model endpoints have throughput limits. Azure OpenAI Provisioned Throughput Units (PTU) give you reserved capacity at a fixed price, but cap out at a hard ceiling. Pay-as-you-go (PAYG) endpoints scale elastically but incur higher per-token costs at high volumes. Without a gateway layer, individual applications can’t know whether PTU capacity is available or saturated. A gateway can make that routing decision automatically, serving from PTU when it has headroom, falling back to PAYG when it doesn’t. That’s a meaningful cost optimization with no changes required to the calling applications.

Cost control: Tokens consumed by one team are costs borne by another team’s budget if you’re centralizing AI spend, which most organizations will do, at least initially. Without per-consumer visibility into token usage, internal cross-charging and showback are impossible. APIM’s token metric policies make this tractable by emitting token consumption data broken down by dimensions such as User ID, Subscription ID, or API product, all of which feed into Application Insights for dashboarding and alerting.

Data security: Routing AI traffic through a managed gateway gives you a single enforcement point for authentication, authorization, and policy. You can validate JWT claims, require subscription keys from API consumers, use Managed Identity to authenticate to Azure OpenAI without exposing credentials, and ensure traffic never leaves your controlled perimeter. Without a gateway, every team builds its own auth story, or more commonly, skips it.

PTU vs. PAYG: Why the Billing Model Shapes Your Architecture

Before we go further, it’s worth spending a moment on the two Azure OpenAI billing models, because they have direct architectural implications.

Provisioned Throughput Units (PTU) give you reserved capacity on a model. You pay a fixed hourly rate regardless of how many tokens you actually consume. The benefits are predictable costs and guaranteed throughput. The risk is waste if your utilization is low, and hard throttling if you exceed the provisioned limit.

Pay-as-you-go (PAYG) charges per token consumed. No upfront commitment, no capacity ceiling, but costs scale linearly with usage and can surprise you if consumption spikes.

Most production AI deployments end up using both: PTU for baseline load, where utilization is predictable, and PAYG as an overflow layer. This makes a load balancer with circuit breaking essential, which we’ll cover in Part 5 of this series.

The same logic applies beyond Azure OpenAI. APIM now supports generic LLM backends via the llm-* policy family, which means you can manage traffic to Mistral, Cohere, LLaMA, and other providers through the same control plane. The diagram below shows this architecture: APIM in the center, with load balancing across PTU and PAYG instances, token metrics flowing to Application Insights, and the full provider landscape behind it.

What This Looks Like in Practice

Let me make this concrete with a scenario I’ve seen play out multiple times.

An organization deploys its first Azure OpenAI service for a conversational agent. A few months later, a second team wants to use AI for content generation. Then a third team builds an internal copilot. Each team provisions its own Azure OpenAI resource, authenticates directly, and manages its own rate limiting. There’s no visibility into combined spend. No shared capacity optimization. No centralized audit trail.

This is the point where someone in finance asks a question that nobody can answer: “How much are we spending on AI, and which team is spending what?”

Centralizing AI traffic through APIM is how you get out of that situation before it becomes a problem. The policy-based approach means you can add governance without changing anything in the calling applications. They call the APIM endpoint, APIM handles the rest.

Azure API Management for AI Workloads: What’s Coming in This Series

The next six posts will go deep on the specific capabilities that make APIM a serious AI control plane:

Part 2 covers authentication and authorization: JWT validation, Managed Identity, and subscription keys.
Part 3 covers the Token Limit policy: enforcing tokens-per-minute limits per consumer.
Part 4 covers the Token Metric policy: emitting usage data for observability and cross-charging.
Part 5 covers load balancing and circuit breaking across PTU and PAYG backends.
Part 6 covers semantic caching: reducing token consumption by serving cached responses for similar prompts.
Part 7 covers APIM’s emerging role as an MCP gateway for agentic AI workloads.

Each post will include the relevant policy XML, real-world sizing guidance, and the architectural decisions behind the patterns.

If you’re building AI-powered applications at scale and you’re not yet routing that traffic through a gateway, the rest of this series is for you.

Cloud Perspectives

Steef-Jan Wiggers

Tag Archives: APIM for AI Workloads Series

Azure API Management as MCP Gateway: Governing Agentic AI Workloads

What MCP Is and Why It Changes the APIM Story

Azure API Management as MCP Gateway: Three Capabilities

Applying Series Policies to Agentic Workloads

Practical Considerations for APIM as MCP Gateway

Azure API Management as MCP Gateway: Closing the Series

Azure API Management Semantic Caching: Cut AI Token Costs with Similarity-Based Responses

How Azure API Management Semantic Caching Works

Tuning the Score Threshold for Azure API Management Semantic Caching

vary-by Scope: Preventing Cache Pollution

Infrastructure Requirements for Azure API Management Semantic Caching

What’s Next in This Azure API Management for AI Series

Azure API Management Token Metric Policy: AI Cost Observability and Cross-Charging

Azure API Management Token Metric Policy: How It Works

Choosing Dimensions for Azure API Management Token Metric Policy

Querying Token Metrics in Application Insights

From Observability to Cross-Charging

AI Gateway Commercial vs Open Source: How to Choose the Right Control Plane

Why the AI Gateway Commercial vs Open Source Choice Matters More Than You Think

The Commercial Options for AI Gateway

The Open Source Options for AI Gateway

AI Gateway Commercial vs Open Source: Five Decision Factors

A Decision Framework for AI Gateway Commercial vs Open Source

Where This Fits in the APIM for AI Workloads Series

Azure API Management Token Limit Policy: Controlling AI Token Consumption Per Consumer

Azure API Management Token Limit Policy: How It Works

Choosing a counter-key for Azure API Management Token Limiting

Sizing Your TPM Limits

Handling 429 Responses in Calling Applications

What’s Next in This Azure API Management for AI Series

Azure API Management for AI: Securing Your AI APIs with Authentication and Authorization

Azure API Management for AI Security: A Three-Layer Model

Layer 1: Subscription Keys

Layer 2: JWT Validation and Claims-Based Authorization

Layer 3: Managed Identity for Backend Authentication

Azure API Management for AI: Putting the Three Layers Together

What’s Next in This Series