This series started with a map and grew into seven pieces. Five layers came first: compute, where load shape picks the service; messaging and orchestration, where two questions replace four product choices; data patterns, where idempotency and the outbox keep a platform correct; governance and identity, where policy and audit become the compliance posture; and observability and FinOps, where behaviour and cost become visible. Then came the lens: from design to demonstrable operation, the shift from “is it built?” to “can we operate it responsibly?”
Every one of those pieces rests on a shared assumption. The system does what you told it to do. You wrote the workflow, you defined the routes, you set the policies, and the platform executes them. That assumption has held for every integration platform I’ve built. Agentic workloads break it. So this capstone asks what changes when the thing making decisions inside your platform is a model, not your code, and why each layer, plus the readiness lens itself, deserves a second look because of it.
The assumption agentic workloads break
Conventional integration is deterministic. A message arrives, a workflow runs its defined steps, a router sends it where the rules say. You can read the code and know what will happen. You can test every path. When something fails, you trace it to a step you wrote.
Agentic workloads replace part of that determinism with a model that decides at runtime. The agent reads context, picks a tool, interprets the result, and chooses the next action. Moreover, it does so differently depending on inputs you didn’t fully anticipate. That’s the point of it: the flexibility is the feature. But it means you can no longer read the code and know what will happen. So the ground under every layer shifts: behavior is no longer exactly what you specified.
None of this argues against agentic workloads. It argues for revisiting each layer with the shift named explicitly. Let’s do that.
Compute: the loop changes the shape of the work
The compute layer sorted workloads by load shape: steady request traffic to App Service, event-driven bursts to Functions or Container Apps. Agentic workloads add a shape that sorting didn’t account for: the loop.
An agent doesn’t process a request and return. It reasons, calls a tool, waits, observes, and reasons again, sometimes for many cycles, before it finishes. That’s neither a clean request-response nor a discrete event. Instead, it’s a long-running loop of unpredictable duration with external calls in the middle. So the compute question changes. You’re no longer asking “steady or bursty” alone. You’re asking how to host something that runs for seconds or minutes, holds state across tool calls, and scales on a dimension concurrent reasoning loops that CPU-and-memory autoscale captures poorly. Container Apps with event-driven scaling often fit better here than App Service, and the orchestration frequently belongs in a workflow engine rather than raw compute.
Messaging and orchestration: the agent is a non-deterministic router
The messaging layer drew a clean line. Deterministic routing rules sent messages where the logic dictated. An agent orchestrating tool calls is, in effect, a router too, but a non-deterministic one. It decides which tool to call from its reading of the context, not from a rule you wrote.
The reliability consequences are real. Delivery guarantees still matter; an agent that triggers a business action still needs that action to occur exactly once, so Service Bus and the idempotency store in the data layer remain as relevant as ever. What changes is predictability. You can’t fully anticipate which actions the agent will trigger, or in what order. Therefore, the orchestration has to stay correct under sequences you didn’t design for. One practical lesson from building these loops applies directly: agent outputs rarely arrive as the clean structures a deterministic step would emit, so you build explicit bridges between agent actions rather than assuming shape.
Data: state and correctness under non-determinism
The data patterns held a platform correct when systems it didn’t control misbehaved. Agentic workloads make those patterns more necessary, not less, and they add one more.
Idempotency matters more because an agent may retry a tool call or repeat an action as it reasons, so the dedup store carries a heavier load. The outbox matters just as much, because an agent-triggered write still has to propagate reliably. Workflow state matters more too, since the reasoning loop is exactly the kind of long-running, restart-surviving process that needs durable state and a correlation ID. And then the new one: conversation and context state. An agent carries context across turns, and that context has to live somewhere durable and queryable, which explains why a flexible document store keeps showing up as the default for agentic conversation state. The access pattern points at the store. Same principle as the map, applied to a new kind of state.
Governance and identity: where the assumptions break hardest
This layer changes most, and I’d insist any integration architect think it through before shipping an agentic workload.
The governance layer secured a deterministic platform. Identity answered who the caller was; policy constrained what the platform could be. Both still matter. However, agentic workloads open a gap that neither fully closes. Identity secures who the agent is. It does not touch what a poisoned tool result or a manipulated retrieved document makes the agent do. Prompt injection rides in through the data the agent requested inside the reasoning loop, downstream of the perimeter check everyone assumes protects them.
So the governance layer needs additions a deterministic platform never required:
Authorization moves per-action: A validated identity at the edge isn’t enough. Each tool call the agent makes needs its own check: is this specific action allowed for this tenant right now? The perimeter check happens once; the risk recurs on every call inside the loop.
Recovery means compensation, not retry: Agent actions have side effects across systems. A failed sequence three actions deep can’t restart from the top; it needs compensating actions to undo what already happened. That’s saga-style thinking, and you design it; it doesn’t emerge.
Containment has to be possible: When an agent misbehaves, you stop it fast, and at more than one layer. Layered containment, from a single configuration flip-up to a full block, turns “contain the agent” from an incident-call debate into a seconds-long operation.
Evaluation becomes a first-class layer: Operational observability tells you the agent is running. It doesn’t tell you the agent’s outputs are quietly degrading. Under the EU AI Act’s oversight and transparency duties, that stops being optional polish and becomes evidence you’re meeting an obligation.
The readiness lens, asked again
The design-to-operation post posed the question that decides go-live: not “is it built?” but “can we operate it safely, recoverably, auditably, and predictably?” Agentic workloads sharpen every word of that sentence.
Safely now includes per-action authorization and containment, because the threat walks in as data. Recoverably now means compensation and sagas, because retry alone can’t undo side effects. Auditably now covers what the agent accessed, which tool it called, why it acted, and what policy constrained it evidence the EU AI Act increasingly expects. And predictably is precisely the property the agent gave up, which is why the surrounding architecture has to supply it instead. The production baseline, the demonstrable-versus-designed test, the three moments of readiness all of it still applies. Each bar sits higher.
The revised framework
The map closes with five questions. For agentic workloads, they hold, and each gains a harder edge. Before an agentic workload goes near a real system, I’d add these:
Can I host a long-running reasoning loop, not just a request or an event? Can my orchestration stay correct when I can’t predict the action sequence? Does my data layer hold conversation state as well as business state, with idempotency doing heavier duty? Is authorisation per-action, not just per-identity? Can I contain a misbehaving agent in seconds? And can I evidence what the agent did, why, and whether its quality held?
Those aren’t different questions from the series. They’re the same layers, asked again under non-determinism, and then held up against the readiness lens one more time.
The shape of it
Each layer in the series meets its agentic stressor: the loop, the non-deterministic router, conversation state, injection past the perimeter, the evaluation question, and the readiness lens. Spanning them all, the workload surrendered predictability, so the architecture supplies it.
Agentic workloads don’t replace the Azure PaaS foundation an integration architect builds on. They stress it. Every layer in this series still includes compute, messaging, data, governance, and observability, but each one now supports a workload that decides for itself at runtime. The compute layer meets the loop. The messaging layer meets a non-deterministic router. The data layer meets conversation state and heavier idempotency. The governance layer meets a threat that walks in through the front door as data. And the readiness lens meets a workload that surrendered predictability, so the architecture has to supply it.
The through-line of the whole series holds here too. The model is the least differentiated part of a production agent. What separates a demo from something you can run against real systems in a regulated industry is the architecture around it: the same layers, asked harder, and proven in operation rather than promised in design. So the foundation was never wasted. It’s exactly what agentic workloads need, applied with the assumptions made explicit.
That’s the series. Start at the Azure PaaS map for the layer-by-layer foundation, take the design-to-operation lens with you as the test, and come back here for what changes when the workload thinks for itself.
That question is harder than it looks, because “we built it” and “we can run it” are different claims. I’ve watched more than one integration platform pass every technical checkpoint and still fall short of production-ready, not because the design missed anything, but because nobody had turned that design into something enforceable, operable, and provable. So this post is about the gap between those two states, and how you close it. It’s the through-line under every layer, and it’s the thing that turns a strong architecture into a platform you can responsibly put load on.
The shift: from “is it built?” to “can we operate it?”
Here’s the single most useful reframe I know for this stage. Stop asking “is the platform technically built?” and start asking “can we operate it safely, recoverably, auditably, and predictably?”
A platform can pass the technical-built question comfortably and fail the operational one completely. The left asks whether components exist and connect; the right asks whether we can see it, triage it, recover it, prove it, and hand it over- this is the question that decides go-live.
Those are not the same question. The first is about whether the components exist and connect. The second is about whether, when something goes wrong at 2 am, someone can see what happened, understand it, recover from it, and prove afterward that they handled it correctly. A platform can pass the first test comfortably and fail the second completely. And the second test is the one that actually determines whether you should go live. So the moment you catch yourself saying “it works,” push on: does it work in a demo, or does it work under a failure you didn’t plan for?
The core move: make the implicit explicit
Most integration platforms at this stage share the same shape. The design is good, and someone has largely written it down. But a lot of what matters lives in documentation, in code that’s still evolving, or in the heads of the people who built it. That works fine while a small team builds the foundation. It stops working the moment the first real production use case lands, because implicit choices become whatever the first team happens to decide.
The fix is a production baseline: an explicit, enforceable statement of what production use demands. It names the mandatory components and patterns, pins down the environment profiles, fixes the security controls and monitoring standards, and settles the recovery agreements and release criteria. Its job is to pull those decisions out of documents and habits and into something the platform enforces, so the first production use case inherits the decisions rather than reinventing them.
Without that baseline, every early integration is free to make its own choices, and you accumulate inconsistency, technical debt, and a future re-platforming you didn’t budget for.
Design versus demonstrable: the recurring gap
The same gap shows up in every layer, and once you see it, you can’t unsee it. On security, the design names Zero Trust, least privilege, and pipeline-driven change, yet permanent broad access rights sit in the environment, quietly contradicting it. Observability design specifies OpenTelemetry and required fields, but the alert rules and dashboards never make it into the infrastructure. And the CI/CD design describes a full release chain with quality gates, while the pipelines really only cover the dev environment.
In each case the design is right and the demonstrable working is missing. That’s the pattern to hunt for when you assess readiness: not “did someone design this?” but “does the platform actually enforce this design somewhere it can prove?” Anything that lives only as intent is a gap, however good the intent.
What operational readiness actually covers
Technical function is necessary but not sufficient. Operational readiness asks a distinct set of questions, and that set decides go-live. In practice, it comes down to whether you can demonstrably answer these:
Can you see what’s happening: monitoring, chain-level tracing, message-level insight? When an incident hits, does someone triage it, and do they have the information they need to do so? For recovery, can the platform handle errors, replay from a known point, and fall back on a real Business Continuity and Disaster Recovery plan rather than an RTO written in a document? Afterward, can you prove what happened through an audit trail, an access log, and change history? On access control, do you separate permanent from elevated rights, and dev from production? For cost, can you attribute it, budget against it, and tier retention? And finally, can you hand it over — have you defined operational ownership, or does every incident route back to the people who built it?
If any of those answers is “only in the design,” please fix it before go-live, not after the first incident teaches you the hard way.
The boundary that decides scale: central versus decentral
The other thing production-readiness has to settle is a boundary, not just a checklist. Most modern integration platforms want value-stream teams to deliver independently within central guardrails. That’s the right ambition. But it only works when you draw the line between central platform ownership and team autonomy deliberately, because that line runs through every layer: API governance, messaging configuration, RBAC, pipeline use, monitoring, error handling, lifecycle, support.
Get the line wrong and you recreate the exact problem the platform set out to solve. The platform team becomes the bottleneck again — the single point through which every change, approval, and incident has to pass. So team autonomy isn’t just a tooling question. It needs explicit ownership agreements, release paths, access models, quality controls, and operational responsibilities. The tooling enables autonomy; the agreements make it safe.
Shared components are where this bites hardest. A shared API gateway, a shared message broker, a shared logging workspace these touch technology, security, governance, operations, cost, and autonomy all at once. They make the platform economical, and they carry the biggest scaling risk. For each one, decide deliberately: why does it stay shared rather than isolated, who owns it, who may change it, how do you monitor it, and how do you attribute its cost? Leave those implicit and the shared component quietly becomes everyone’s dependency and no one’s responsibility.
Readiness isn’t one moment — it’s three
The last reframe worth making: “ready” isn’t a single bar. It’s three different bars at three different moments, and conflating them is how platforms either over-build early or under-prepare for scale.
“Ready” is three different bars. First go-live demands a production baseline and demonstrable operability; first team onboarding tests whether the federated model works in practice; scaling to many teams demands repeatability. Naming the moment stops you over-building early or under-preparing for scale.
First go-live. The bar here is the minimum production baseline and demonstrable operational readiness. Not every capability has to be complete, but the ones that are preconditions for running safely in production do. This is where the baseline, the recovery plan, and the release criteria have to be real.
First team onboarding. The bar shifts to whether the federated model actually works in practice. Can one real team deliver independently within the central guardrails, without quality, security, or consistency buckling under the first real use? This is a practice test, and it’s better, even, to let some things get concrete here rather than designing them fully in the abstract.
Scaling to many teams. Now the bar is repeatability. Anything that worked at one team through direct conversation now has to become standardised, documented, and reproducible: lifecycle policy, cost allocation, onboarding, support model, versioning, exception handling. Direct alignment doesn’t scale; product-steering does.
Naming which moment a given concern belongs to is half the battle. It stops you from demanding scale-grade rigour before first go-live, and from discovering at team five that nobody built the repeatable version.
Where this thinking gets over-applied
Consistent with the series, the honesty section. “Production baseline” thinking is right, but it can tip into paralysis.
Not everything has to be complete before first go-live. The three-moments split exists precisely so you don’t. Demanding full lifecycle policy, mature FinOps, and a complete federation model before a single use case runs is how a platform never ships. Match the rigour to the moment.
A baseline that only flags is a baseline that gets ignored. The whole point of the production baseline is that the platform enforces it. A pile of documented-but-unenforced standards manufactures the appearance of readiness without the substance, which is more dangerous than an honest gap, because it invites false confidence.
You can make a decision deliberately without making it central. Drawing the central-versus-decentral line carefully doesn’t mean pulling everything central. Sometimes the deliberate call is “this is team-owned,” and recording that reasoning is the point, not the direction.
The shape of it
For an integration architect, production-readiness isn’t a technical checkpoint; it’s the shift from a platform that’s built to one you can operate responsibly. Make the implicit explicit in a production baseline. Hunt the gap between what’s designed and what’s demonstrable. Answer the operational-readiness questions before go-live, not after. Draw the central-versus-decentral line deliberately, especially for shared components. And treat “ready” as three moments, not one. Do that, and the layers from this series stop being a good architecture on paper and become a platform you can actually run.
This is the lens that ties the series together. The Azure PaaS map has the layer-by-layer foundation; this post is the question you hold every layer up against before you put production load on it.
In the Azure PaaS map post, the data layer got one paragraph and a rule: pick by access pattern, not by which service feels modern. That rule holds. But it’s also where most write-ups stop: SQL for relational integrity, Cosmos DB for scale, and Redis in front; for an integration architect, that’s the least interesting part of the story.
The interesting part is what the data layer has to do that’s specific to integration. Messages arrive twice. Workflows run for hours and need somewhere to keep their state. A write to your database and a publish to a queue have to succeed or fail together. So this post skips the service comparison and covers the patterns instead. Azure data patterns for integration are less about which store you pick and more about how you use it.
Why integration data is different
A typical application owns its data. It writes, it reads, it controls the whole path. Integration doesn’t work that way. Instead, integration sits between systems it doesn’t own, reacting to events it didn’t originate, and it has to stay correct when those systems misbehave.
That changes what the data layer is for. It’s no longer just persistence. It becomes the place where you enforce correctness that the messaging layer can’t guarantee on its own. Three patterns come up again and again. Let’s take them in turn.
Pattern 1: Idempotency stores
Here’s the problem. At-least-once delivery is the norm for most messaging systems, including Service Bus. So the same message can arrive twice after a retry, a redelivery, or a consumer crash-and-restart. Process it twice, and you’ve charged the card twice or created two orders. That’s not a rare edge case. In a busy integration platform, it’s a Tuesday.
At-least-once delivery means the same message can arrive twice. Check the ID against a store first; skip if seen; record and process once if not.
The fix is an idempotency store. Before you process a message, you check whether you’ve seen its ID before. If you have, you skip it. If you haven’t, you record the ID and proceed. As a result, duplicate deliveries become harmless.
The design questions that matter:
Where does the key come from? Ideally, the source system supplies a stable business key, an order ID, and a transaction reference. Failing that, a hash of the message content works, though it’s more fragile.
Where do you store it? This is a high-frequency, low-latency lookup on a single key. Therefore Cosmos DB or Redis fit well, and a relational table works too if the volume is modest. The access pattern points at the store, exactly as the map post argued.
How long do you keep it? Retention has to outlast the longest possible redelivery window. Too short, and a late duplicate slips through. So set a TTL that comfortably exceeds your retry and dead-letter timelines, then expire old keys automatically.
The honest note: idempotency at the store isn’t the same as an idempotent operation. If the downstream side effect isn’t itself safe to repeat, the store only narrows the window; it doesn’t close it. Design the operation to tolerate retries wherever you can.
Pattern 2: The outbox pattern
This one solves the dual-write problem, and the dual-write problem is subtle enough that plenty of teams ship it broken.
Picture a handler that does two things. It writes a record to the database, and it publishes an event to a queue. Both must happen, or neither. But they’re two separate systems, so there’s no shared transaction. Write succeeds, publish fails; now the database and the downstream world disagree. Publish succeeds, write fails; now you’ve announced something that didn’t happen.
The business record and the outbox event commit in a single database transaction, so they succeed together. A separate publisher then tails the outbox and publishes each event, giving at-least-once delivery downstream.
The outbox pattern closes the gap. Instead of publishing directly, you write the event into an “outbox” table in the same database transaction as your business record. Because they share one transaction, they commit together or not at all. Then a separate process reads the outbox and publishes the events, marking each one done as it goes.
A few things fall out of this design:
The database becomes the source of truth for what should be published. If the publisher crashes mid-run, it restarts and picks up where it left off. Nothing is lost, because nothing left the database until it was safely committed.
Publishing becomes at-least-once. The publisher might send an event, crash before marking it done, and send it again on restart. So the consumer on the other end needs, you guessed it, an idempotency store. The two patterns work together.
A change-feed makes it cleaner. Cosmos DB’s change feed, or a similar mechanism, lets the publisher tail committed changes rather than poll a table. That reduces latency and load, though a simple polling publisher is perfectly fine to start.
The trade-off is honest latency. The outbox adds a hop between commit and publish. For most integration workloads that’s a few seconds at most, and well worth it for the correctness guarantee. But if you need genuinely instant propagation, the outbox isn’t your pattern.
Pattern 3: State for long-running workflows
Synchronous request-response keeps its state in memory for the length of a call. Integration workflows don’t have that luxury. A process can span minutes, hours, or days while waiting for approval, a batch window, or an external callback. That state has to live somewhere durable, because the compute running it will scale, restart, and move underneath it.
So where does workflow state go? It depends on who’s orchestrating.
The compute running a workflow scales, restarts, and moves, so the state has to live somewhere durable. A workflow engine manages it for you; hand-rolled orchestration needs an explicit store. Either way, a correlation ID reconnects a callback to the right in-flight instance.
Logic Apps and Durable Functions manage their own state: Both persist workflow state for you; that’s a large part of why they exist. Durable Functions keeps it in a storage backend; Standard Logic Apps keeps it in its own runtime store. In these cases, you rarely touch the state directly, but you should know it’s there and know that it’s what makes the workflow survive a restart.
Hand-rolled orchestration needs an explicit store: When you’re coordinating steps in your own code rather than a workflow engine, you own the state. A document store like Cosmos DB fits well here: one document per workflow instance, updated as the process advances through its steps. The flexible schema helps, because a workflow’s state shape often evolves as you add steps.
Correlation is the piece people forget: Long-running workflows wait for things to come back, and when a callback arrives, you have to match it to the right in-flight instance. That means a correlation ID, stored with the instance and carried on every outbound call. Without it, you have durable state you can’t reconnect to the event that needs it.
Where these patterns are the wrong answer
Consistent with the rest of the series, the honesty section. Patterns solve problems, and applying them where the problem doesn’t exist adds cost.
Skip the idempotency store when the operation is naturally idempotent: Setting a status to “shipped” twice changes nothing. If every side effect is already safe to repeat, a dedup store is machinery you don’t need.
Skip the outbox when you don’t dual-write: If a handler only writes to the database, or only publishes, there’s no gap to close. The outbox earns its keep specifically when one commit must produce one publish.
Skip explicit state stores when a workflow engine already owns the state: Standing up your own Cosmos-backed state store next to Durable Functions duplicates what the runtime already gives you. Reach for the explicit store only when you’re orchestrating by hand.
The shape of it
For an integration architect, the data layer isn’t mainly a choice between SQL and Cosmos. That choice matters, but the access pattern usually makes it for you. The real work is the patterns that keep an integration platform correct when systems it doesn’t control misbehave. So an idempotency store absorbs duplicate deliveries. An outbox makes both a write and a publish succeed. A durable state store lets a workflow outlive the compute running it. Get those right, and the underlying store SQL, Cosmos, and Redis become implementation details rather than the headline.
Want the layer this sits inside? The Azure PaaS map puts data in context against compute, integration, and governance, and walks the five-question framework across all of them. And the messaging and orchestration post covers the delivery guarantees these patterns lean on.
The Citadel APIM gateway policies on Azure are doing the heavy lifting silently in every post in this series. We deployed the hub, connected a tool-calling agent, added conversation persistence, and demonstrated the kill switch. Throughout all of that, every chat completion request passed through these policies. They counted every token. They produced all usage telemetry. This post opens the hood and examines all five Citadel APIM gateway policy layers in Azure token rate limiting, semantic caching, content safety routing, cost attribution, and PII redaction with the actual policy XML, live demonstration results, and honest debugging sessions included.
Where the Citadel APIM Gateway Policies Live in Azure
The Citadel hub deploys two types of policies.
API-level policies apply to all operations on the Azure OpenAI API for every chat completion, embedding, and batch request. These contain the token-limiting, usage-tracking, and routing logic.
Policy fragments serve as reusable blocks that you include by reference. The hub uses fragments for AAD authorization (aad-auth), load balancing (openai-backend-pool), and dynamic throttling (dynamic-throttling-assignment).
In the portal, find them at apim-wpvlimv4ngkns → APIs → Azure OpenAI Service API → All operations → Policies.
The full policy XML is also in the repo at github.com/Azure-Samples/ai-hub-gateway-solution-accelerator under infra/modules/apim/policies/.
APIM natively supports rate limiting on the number of requests per time window. The hub repurposes this to manage capacity based on tokens — the azure-openai-token-limit policy limits how many tokens per minute a subscription can consume, using a counter keyed to the APIM subscription.
counter-key=”@(context.Subscription.Id)” — the counter is per APIM subscription, so each spoke gets its own independent token budget.
tokens-per-minute=”10000″ — 10,000 TPM limit per subscription; adjust per spoke based on workload.
estimate-prompt-tokens=”true” — APIM estimates prompt tokens before the response arrives, enabling proactive rate limiting rather than post-hoc tracking.
tokens-consumed-variable-name=”TotalConsumedTokens” — stores the running count for use by downstream policies (cost attribution reads this variable).
Demonstration
The Citadel hub actually applies two independent token limit policies at different scopes, and tracing the real behaviour reveals an important lesson about how APIM evaluates them.
Scope 1 — subscription-level, per deployment. The product policy on oai-retail-assistant assigns a different tokens-per-minute value based on the targeted deployment.
Both policies execute on every request, each maintaining its own counter. Whichever limit you hit first governs the response, regardless of which one you configured.
Live test — three rapid requests against the chat deployment with the subscription-level limit set to 50 TPM:
REQUEST 2 — Status: 429, Body: “Token limit is exceeded. Try again in 51 seconds.”
REQUEST 3 — Status: 429, Body: “Token limit is exceeded. Try again in 51 seconds.”
Notice that Remaining-Tokens: 14940 on the first request comes from the product-level counter (15000 TPM, barely touched). The 429 on request 2, however, comes from the subscription-level counter for the chat deployment, which only allows 50 tokens per minute, exhausted after a single 15-token call. The visible remaining-tokens header reflects whichever counter last wrote to it, which can be misleading if you assume there is only one limit in play.
When multiple azure-openai-token-limit policy elements exist at the API level, product level, and per deployment in a choose block, they are all evaluated for each request. In addition, the most restrictive policy that triggers first decides the outcome. Furthermore, to debug rate limit issues, check all policy scopes: API-level, product-level, and deployment-specific branches. A Named Value in one scope doesn’t affect a hardcoded limit in another.
Citadel APIM Gateway Policy 2 — Semantic Caching
What It Does
Semantic caching intercepts requests before they reach Azure OpenAI and checks for previously answered semantically similar questions. If it finds a match above the configured similarity threshold, it returns the cached response immediately, using zero tokens and ensuring near-zero latency.
The Policy XML
The lookup and store directives belong in different policy sections — lookup runs on the inbound request, store runs on the outbound response after a successful call.
score-threshold=”0.8″ — similarity must be 80% or higher for a cache hit. Lower values increase hit rate but risk returning mismatched responses.
embeddings-backend-id=”openai-backend-0″ — points to the backend that hosts the embeddings deployment used for cache lookup. The embeddings model itself is configured on the backend, not as a policy attribute — embeddings-model is not a valid attribute on this policy element and will fail schema validation if added.
embeddings-backend-auth=”system-assigned” — the embeddings backend call is authenticated via the APIM instance’s system-assigned managed identity.
ignore-system-messages=”true” — only user messages are used for cache key generation, not system prompts.
max-message-count=”5″ — only the last 5 messages in a conversation are used for cache lookup.
duration=”600″ — cached responses expire after 10 minutes.
Demonstration
First request — cache miss:
pythonagent.py
# Question: What is the weather like in Stockholm right now?
# Response time: 1.2 seconds
Second request — cache hit:
python agent.py
# Question: What is the weather in Stockholm today?
# Response time: 45ms
The second question resembles the first semantically, but it’s not an exact match. The 0.8 threshold identifies it as a cache hit, allowing the system to return the response from the cache in 45ms without making an Azure OpenAI call or using any tokens. For a conversational agent that often encounters similar questions, semantic caching can lower token consumption by 20–40%.
Production Consideration
The cache shares its data across all subscribers to the same APIM product. If two spokes utilize the same APIM gateway, Spoke A’s cached response can serve Spoke B for a similar inquiry. To protect sensitive data, configure the cache key to be scoped per subscription by adding @(context.Subscription.Id).
Pitfall: Policy Section Placement and Schema Validation
When adding semantic caching for the first time, you may encounter two common schema errors. First, placing azure-openai-semantic-cache-store in the inbound section, along with the lookup policy, results in the error: “Policy is not allowed in this section.” Second, you’ll find that embeddings-model is not a declared attribute on azure-openai-semantic-cache-lookup. Instead of being specified directly in the policy, the embeddings deployment is retrieved from the backend referenced by embeddings-backend-id. You can quickly identify both errors when you save the policy in the portal, which provides the fastest validation of your XML before deployment.
The hub routes every request through Azure AI Content Safety before it reaches Azure OpenAI. It inspects both the user prompt and the model response for harmful content in four categories: hate, self-harm, sexual, and violence. The system blocks any requests or responses that exceed the configured severity thresholds.
backend-id=”content-safety-backend” — routes to the cog-consafety-wpvlimv4ngkns Azure AI Content Safety instance deployed in the hub.
shield-prompt=”true” — enables Prompt Shields, which additionally detects jailbreak attempts and prompt injection on top of standard content categories.
categories output-type=”FourSeverityLevels” — selects the four-level severity scale (0, 2, 4, 6) rather than the eight-level scale; each category child element sets its own threshold independently.
category name=”…” threshold=”2″ — one element per category (Hate, SelfHarm, Sexual, Violence), each can have a different threshold. A threshold of 2 blocks low severity and above; omitting a category leaves it unchecked.
You can reference custom Azure AI Content Safety blocklist IDs in an optional blocklists element to always block organization-specific terms, regardless of their severity scoring.
Pitfall: Element Name and Schema
The element is llm-content-safety, not azure-content-safety — using the wrong name produces “There is no policy matching element name” on save. The schema is also structural rather than attribute-based: categories are child category elements inside a categories block, not a flat comma-separated categories=”…” attribute. Both errors surface immediately in the portal policy editor, which validates the XML before allowing a save.
Demonstration
Normal request — content safety passes silently:
pythonagent.py
# Question: What is the weather like in Stockholm?
# Answer: The weather in Stockholm is overcast...
The Policy
The llm-content-safety policy does not add a confirmation header on a passing request — the absence of a block response is the only signal that the prompt and response cleared all four category thresholds. To confirm the policy actually ran, check Application Insights:
requests
|wheretimestamp> ago(10m)
|where resultCode =="200"
| project timestamp, name, resultCode, duration
|orderbytimestampdesc
A 200 with normal duration confirms the request passed through llm-content-safety without being blocked.
Harmful content
Blocked request — harmful content detected:
# Modify agent.py to send a harmful prompt, then run:
pythonagent.py
# openai.BadRequestError: Error code: 400
# 'Request failed content safety check.'
Requests block at the gateway before they reach Azure OpenAI, ensuring that these blocked requests consume no tokens. In Application Insights, you observe the block as a 400 response with near-zero duration, reflecting the same signature pattern seen with the kill switch layers mentioned in the previous post.
Every request that completes successfully generates a usage event sent to Azure Event Hub. A Logic App deployed in the hub consumes these events and writes structured documents to the Cosmos DB ai-usage-container. Each document contains token counts, model version, gateway region, APIM subscription name, and timestamp, giving you per-subscription cost attribution without any agent-side code changes.
The Policy XML
The hub deploys three named loggers, visible via az rest against the APIM management API: appinsights-logger for Application Insights telemetry, usage-eventhub-logger for the cost attribution pipeline described here, and a separate pii-usage-eventhub-logger for PII-specific event logging tied to the redaction policy described later in this post. The logger-id attribute on log-to-eventhub must match one of these exactly — using a placeholder or guessed name produces “Logger not found” when saving the policy.
var usage = response.Body.As<JObject>(true)?["usage"];
return new JObject(
new JProperty("id", context.Response.Headers.GetValueOrDefault("x-request-id", Guid.NewGuid().ToString())),
new JProperty("timestamp", DateTime.UtcNow.ToString("o")),
new JProperty("subscriptionId", context.Subscription?.Id),
new JProperty("productName", context.Product?.Name),
new JProperty("targetService", "chat.completion"),
new JProperty("model", response.Body.As<JObject>(true)?["model"]?.ToString()),
new JProperty("gatewayName", context.Deployment.ServiceName),
new JProperty("gatewayRegion", context.Deployment.Region),
new JProperty("RequestIp", request.IpAddress),
new JProperty("promptTokens", usage?["prompt_tokens"]?.ToObject<int>() ?? 0),
new JProperty("responseTokens", usage?["completion_tokens"]?.ToObject<int>() ?? 0),
new JProperty("totalTokens", usage?["total_tokens"]?.ToObject<int>() ?? 0),
new JProperty("backendId", context.Variables.GetValueOrDefault<string>("backendId")),
new JProperty("deploymentName", context.Request.MatchedParameters["deployment-id"])
).ToString();
}
</log-to-eventhub>
Key fields:
context.Subscription?.Id and context.Product?.Name — the APIM subscription and product name used for the request. In a multi-spoke setup, each spoke has its own APIM product making cost attribution per initiative automatic.
usage?[“prompt_tokens”] / usage?[“completion_tokens”] — extracted directly from the Azure OpenAI response body.
context.Deployment.Region — the gateway region (Sweden Central) for data residency auditing.
context.Variables.GetValueOrDefault<string>(“backendId”) — which Azure OpenAI backend served the request, critical for multi-region deployments.
Demonstration
Run the agent:
python agent.py
Then query the hub Cosmos DB in the portal (cosmos-wpvlimv4ngkns → Data Explorer → ai-usage-container → Items):
SELECT TOP 5 c.timestamp, c.productName, c.model,
c.promptTokens, c.responseTokens, c.totalTokens,
c.gatewayRegion, c.deploymentName
FROM c ORDERBY c._ts DESC
A real document from the deployed hub looks like this:
{
"timestamp":"6/30/2026 11:26:49 AM",
"productName":"OAI-HR-Assistant",
"model":"gpt-4o-2024-11-20",
"promptTokens":93,
"responseTokens":56,
"totalTokens":149,
"gatewayRegion":"Sweden Central",
"deploymentName":"chat"
}
Each agent run produces two documents, one for the tool decision call and one for the synthesis call. The totalTokens across both documents is the true per-conversation cost. The productName field maps directly to the APIM product the calling subscription belongs to, in this example OAI-HR-Assistant, distinct from any other product sharing the same hub.
FinOps in Practice
In production with multiple spokes, filter by productName to get per-initiative cost:
SELECT c.productName,
SUM(c.totalTokens)as totalTokens,
COUNT(1)as requestCount
FROM c
WHERE c.timestamp >="2026-06-01T00:00:00Z"
GROUPBY c.productName
Run against the live hub, this returns a clean per-product summary:
[
{
"productName":"Portal-Admin",
"totalTokens":3865,
"requestCount":32
},
{
"productName":"OAI-HR-Assistant",
"totalTokens":3035,
"requestCount":31
}
]
Two distinct products, each with an independently aggregated token total and request count, no additional instrumentation required beyond the log-to-eventhub policy already in place. This is your direct FinOps input per AI initiative, per month, ready to feed into a PowerBI report or a monthly cost allocation process.
Citadel APIM Gateway Policy 5 — PII Redaction
What It Does
The hub detects personally identifiable information — names, email addresses, phone numbers, IBAN numbers, and other entity types — in conversation content and logs a redacted version to a dedicated Event Hub logger. The original request still reaches Azure OpenAI unmodified; only the logged version is clean.
There Is No Built-In Policy Element for This
The blog draft for this post originally referenced an azure-openai-pii-removal-logging policy element, on the assumption that APIM ships a built-in PII redaction policy the same way it ships azure-openai-token-limit and azure-openai-semantic-cache-lookup. It does not. Saving a policy referencing that element name fails immediately with “There is no policy matching element name,” the same class of error encountered earlier with azure-content-safety versus the correct llm-content-safety.
Unlike content safety, however, there is no equivalent built-in alternative for PII detection at the time of writing. The hub’s pii-usage-eventhub-logger, visible alongside usage-eventhub-logger and appinsights-logger when listing loggers via the APIM management API, exists as infrastructure for this purpose, but the policy that populates it has to be built explicitly using send-request to call Azure AI Language Service directly, then log-to-eventhub to write the result.
The Policy XML
This belongs entirely in outbound, immediately after the cost attribution log-to-eventhub block. PII detection runs on the response side rather than inbound because the goal is to log a redacted record of the conversation, not to block or alter the request itself, context.Request.Body remains accessible in the outbound pipeline, so the original user message can still be analysed at this stage.
PII detection — outbound section, after cost attribution:
var piiResult = ((IResponse)context.Variables["piiDetectionResponse"]).Body.As<JObject>(true);
var redactedText = piiResult?["results"]?["documents"]?[0]?["redactedText"]?.ToString() ?? "[pii-detection-unavailable]";
var entityCount = piiResult?["results"]?["documents"]?[0]?["entities"]?.Count() ?? 0;
return new JObject(
new JProperty("timestamp", DateTime.UtcNow.ToString("o")),
new JProperty("subscriptionId", context.Subscription?.Id),
new JProperty("productName", context.Product?.Name),
new JProperty("redactedText", redactedText),
new JProperty("piiEntitiesFound", entityCount)
).ToString();
}
</log-to-eventhub>
Line by line:
send-request mode=”new” — fires an independent outbound call to Azure AI Language Service rather than reusing the current request/response context. ignore-error=”true” means a Language Service outage does not fail the agent’s actual response, only the PII logging step is skipped.
requestBody[“messages”]?.Last?[“content”] — extracts the most recent user message from the original request body for analysis, since that is where PII is most likely to appear.
The Language Service PiiEntityRecognition endpoint returns both a redactedText field (PII replaced with asterisks) and an entities array listing every detected entity and its category.
log-to-eventhub logger-id=”pii-usage-eventhub-logger” — writes the redacted text and entity count to the dedicated PII logger, kept separate from the general usage logger so PII-related audit records can be access-controlled independently.
Two Named Values must exist before this policy validates:
azapimnvcreate\
--resource-grouprg-ai-hub-gateway-dev\
--service-nameapim-wpvlimv4ngkns\
--named-value-idlanguage-service-name\
--display-name"language-service-name"\
--value"cog-language-wpvlimv4ngkns"\
--secretfalse
azapimnvcreate\
--resource-grouprg-ai-hub-gateway-dev\
--service-nameapim-wpvlimv4ngkns\
--named-value-idlanguage-service-key\
--display-name"language-service-key"\
--value"<your-language-service-key>"\
--secrettrue
Retrieve the Language Service key:
azcognitiveservicesaccountkeyslist\
--namecog-language-wpvlimv4ngkns\
--resource-grouprg-ai-hub-gateway-dev\
--querykey1-otsv
Demonstration
Request containing PII:
question = "What is the weather near Jan Janssen who lives at Keizersgracht 123 Amsterdam?"
First attempt, the policy ran but logged the wrong data. The send-request call to Azure AI Language Service fired correctly and returned 200 every time, confirmed via Application Insights dependency tracking:
Yet the documents landing in pii-usage-container showed no redactedText field at all, only promptTokens, gatewayName, backendId, and the rest of the cost attribution schema:
{
"id":"30db8c4b-ce45-4695-9078-e357405845bc",
"subscriptionId":"oai-hr-assistant-sub-01",
"productName":"OAI-HR-Assistant",
"model":"gpt-4o-2024-11-20",
"promptTokens":109,
"responseTokens":21,
"gatewayName":"apim-wpvlimv4ngkns.azure-api.net",
"backendId":"openai-backend-0"
}
The cause turned out to be a copy-paste error in the log-to-eventhub block itself. The comment above it correctly read “PII detection and redacted logging,” but the JObject construction inside was a verbatim duplicate of the cost attribution payload from usage-eventhub-logger, it never referenced context.Variables[“piiDetectionResponse”] at all. The Language Service call succeeded and its result sat in a context variable, completely unused, while the logger faithfully wrote the wrong document every time. Every dependency call returned 200; every Cosmos DB write succeeded; nothing in the telemetry indicated a problem. The only way to catch it was reading the document schema in Cosmos DB and noticing it matched the usage container rather than containing redacted text.
The fix was correcting the log-to-eventhub body to actually read the stored piiDetectionResponse variable:
var piiResult = ((IResponse)context.Variables["piiDetectionResponse"]).Body.As<JObject>(true);
var redactedText = piiResult?["results"]?["documents"]?[0]?["redactedText"]?.ToString() ?? "[pii-detection-unavailable]";
var entityCount = piiResult?["results"]?["documents"]?[0]?["entities"]?.Count() ?? 0;
return new JObject(
new JProperty("id", context.Response.Headers.GetValueOrDefault("x-request-id", Guid.NewGuid().ToString())),
new JProperty("timestamp", DateTime.UtcNow.ToString("o")),
new JProperty("subscriptionId", context.Subscription?.Id),
new JProperty("productName", context.Product?.Name),
new JProperty("redactedText", redactedText),
new JProperty("piiEntitiesFound", entityCount)
).ToString();
}</log-to-eventhub>
After the fix, the same question produced the correct redacted record:
{
"id":"419321ce-0544-4908-a2ac-9ce10f80aaba",
"timestamp":"2026-06-30T13:12:24.8989851Z",
"subscriptionId":"oai-hr-assistant-sub-01",
"productName":"OAI-HR-Assistant",
"redactedText":"What is the weather near *********** who lives at ***************************?",
"piiEntitiesFound":2
}
Two entities detected and redacted, the person’s name and the street address, while the sentence structure remains intact for log readability. A second request, where the agent’s tool call itself failed to resolve the address, still produced a correctly redacted record of the error message:
{
"redactedText":"{\"error\": \"Location '***************************' not found.\"}",
"piiEntitiesFound":1
}
This confirms PII redaction applies consistently regardless of whether the underlying tool call succeeds, the policy operates on the original user message, independent of how the agent’s downstream logic handles it.
In Cosmos DB usage documents, unaffected: the cost attribution document written by usage-eventhub-logger still contains only token counts and metadata, never request body content, so PII redaction applies exclusively to the dedicated pii-usage-container stream.
Pitfall: A Logger Existing Does Not Mean It Logs the Right Thing
The most instructive failure in this section was not a missing policy element or a schema validation error, it was a policy that validated, deployed, and executed successfully while silently logging the wrong payload. Every signal that normally indicates “this is working” was green: the send-request dependency call returned 200, the log-to-eventhub write succeeded, and documents appeared in Cosmos DB on schedule. The only way to catch the bug was to inspect the actual field names in the logged document and notice they matched a different policy’s output entirely. When wiring up custom logging policies, always verify the logged document shape directly in the data store, a successful HTTP status code on a dependency call says nothing about whether its result was ever used downstream.
Pitfall: Two send-request Round Trips Add Latency
Every request now makes an additional outbound call to Azure AI Language Service before the agent’s response is returned to the caller, since send-request blocks until it completes (or times out at 10 seconds with ignore-error=”true”). For latency-sensitive workloads, consider moving PII detection to an asynchronous pattern, log raw request IDs to Event Hub immediately, then run PII detection as a separate downstream process reading from the Event Hub stream, rather than inline in the request path.
The Complete Policy Execution Order
Understanding the order in which policies execute is critical for troubleshooting. APIM processes policies in this sequence.
This order means a request blocked by the kill switch never reaches token rate limiting, content safety, or Azure OpenAI. A request blocked by content safety never reaches cost attribution or PII logging, no usage document and no redacted-text document are created for blocked requests.
Validating All Five Policies in Application Insights
Use this KQL query to see the policy execution evidence in one view:
tokensConsumed — running token count from rate limiter.
remainingTokens — remaining budget for this subscription.
There is no dedicated content safety column because llm-content-safety does not emit a custom telemetry property, its outcome is entirely reflected in resultCode. A 400 with near-zero duration is the signature of a content safety block, the same pattern used to identify kill switch activations in the previous post.
For PII detection specifically, dependency tracking shows whether the Language Service call fired:
dependencies
|wheretimestamp> ago(24h)
|where target contains "cognitiveservices"and name contains "analyze-text"
A 200 here only confirms the call succeeded, it does not confirm the result was logged correctly downstream. Cross-check against the actual documents in pii-usage-container to verify the redactedText field is present and populated, not just that the dependency call returned successfully.
Pitfalls Summary
Token rate limiting doesn’t work for streaming. Fix: use non-streaming endpoints for accurate token counting.
Semantic cache returns stale weather data. Fix: reduce duration to 60 seconds for time-sensitive tool calls.
Content safety blocks valid medical terminology. Fix: raise threshold to 4 for healthcare-specific deployments and validate against your use case.
azure-content-safety element does not exist. Fix: use llm-content-safety with categories and category child elements, not flat attributes.
azure-openai-pii-removal-logging element does not exist. Fix: no built-in policy exists, implement via send-request to Language Service plus a custom log-to-eventhub.
The logger successfully writes but records incorrect data. To fix this, a 200 status on the send-request and a successful write to Event Hub do not guarantee that the result was actually used. Instead, verify the logged document schema directly in Cosmos DB.
The system misses redacting Dutch-language names. To fix this issue, optimize the analyze-text request by setting the language to “nl” and testing it with representative Dutch inputs, instead of using “en,” which is optimized for English.
Cost attribution missing for some requests. Fix: check Event Hub to Logic App pipeline; ingestion lag of 2 to 5 minutes is normal.
To ensure high compliance in production, always surface failures in content safety and PII detection instead of silently bypassing them with the on-error-action or ignore-error=”true” settings.
Conclusion
The five APIM policies in the Citadel hub, token rate limiting, semantic caching, content safety, cost attribution, and PII redaction, collectively implement enterprise AI governance at the gateway layer. None of them require changes to agent code. None of them require spoke involvement. All of them apply to every request from every agent that routes through the hub, regardless of which spoke deployed it.
Furthermore, they compose cleanly: a request that hits the semantic cache never reaches token rate limiting, content safety, or Azure OpenAI. As a result, cached responses consume zero tokens, zero content safety budget, and zero Azure OpenAI quota. Similarly, a request blocked by content safety produces no cost attribution event and no PII redaction record.
This composability is what makes the Citadel APIM hub a governance layer rather than just a proxy. The policies work together, in a defined order, to enforce the enterprise AI control plane pattern across every governed workload, though, as the PII redaction section makes clear, composability and correct execution are not the same thing. A policy chain can validate, deploy, and run green across every dependency call while still logging the wrong data. The only reliable verification is checking the actual data that lands in the store, not the status codes along the way.
In the Azure PaaS map post, the integration layer got a single paragraph. It named four services: Logic Apps, API Management, Service Bus, and Event Grid, and moved on. This post takes Azure messaging and orchestration apart into the decisions underneath that paragraph.
I won’t tour features here. Two of these services already have their own deep series on this blog, so re-covering them would waste your time. Instead, I’ll stay at the decision layer. When do you reach for which? And why do teams so often reach wrong? Those are the questions that actually cost you in production.
Azure messaging and orchestration: two axes decide almost everything
Four services sound like four choices. In practice, though, only two questions matter, and they cut across the whole layer.
You don’t choose between four services; you answer two questions: messaging or orchestration, then failure cost or ownership. The service falls out from there, with API Management governing the front.
First: is this messaging or orchestration? Messaging moves events and data between systems. Orchestration coordinates a multi-step process toward an outcome. The two look similar on a whiteboard, but they fail differently, scale differently, and belong to different services. So separate them before anything else.
Second: what does failure cost, and what shape is the work? Once you know whether you’re moving messages or coordinating steps, the follow-up question splits the choice further. For messaging, the cost of a lost message decides it. For orchestration, the complexity of the flow and the team who owns it decide it.
Get those two axes clear, and the service almost picks itself. Skip them, and you end up with Event Grid where you needed guarantees, or a Logic App doing work that belonged in code.
Messaging: Service Bus vs Event Grid
Both move things between systems. That’s where the similarity ends.
Service Bus is the durable, ordered, transactional backbone. Reach for it when delivery has to be guaranteed. It gives you sessions for ordered processing, dead-lettering for messages that can’t be handled, and transactional handling across multiple operations. Topics and subscriptions add pub/sub without a separate broker. So Service Bus fits business messages: an order, a payment, a claim, where losing one is an incident.
Event Grid is a lightweight, high-volume router. It broadcasts events to whoever cares: resource state changes, custom application events, and telemetry. It’s built for throughput and fire-and-forget delivery, not guaranteed processing. Therefore, it fits notifications and reactive triggers, where a missed event is a shrug rather than a page.
Here’s the rule of thumb I give teams new to Azure messaging. If losing a message would be a business incident, it belongs on Service Bus. If losing it would just mean a missed notification, Event Grid is fine.
And often you use both. A common pattern pairs them: Event Grid fans out a notification, and a subscriber drops a durable message onto Service Bus for guaranteed processing. That way you get Event Grid’s reach and Service Bus’s reliability in one flow, each doing the job it’s good at.
Event Grid fans an event out to multiple subscribers; the one that needs a guarantee drops a durable message onto Service Bus for ordered, dead-lettered processing.
The honest note, though, is that this is where teams get burned. Event Grid looks simpler, so teams default to it. Then, weeks later, they discover the workload actually needed ordering or delivery guarantees. Now they’re bolting reliability onto a service that was never designed for it. So decide on the message-loss cost first, before the “which feels easier” instinct takes over.
Orchestration: Logic Apps vs code
Messaging moves things. Orchestration coordinates them. The decision here isn’t about reliability; it’s about complexity and ownership.
Logic Apps is the designer-first route. It shines when you need enterprise connectors SAP, IBM MQ, mainframe hosts, the long tail of line-of-business systems without a modern REST API. Standard Logic Apps also closes the old gaps that made it hard in regulated environments: VNet integration, built-in state, per-workflow scaling. So for a workflow that a less code-heavy team will own and maintain, Logic Apps is often the right call even when a Function would be more elegant.
Code is the route once complexity climbs.Designer workflows are fast to build and easy to read at first. Past a certain size, though, they get hard to reason about and harder to code-review. In my experience, the practical ceiling sits around a dozen actions with a couple of branches. Beyond that, do one of two things. Either decompose the workflow into smaller ones, or move the logic into a Function where a proper language and real tests take over.
The deciding questions, then, are simple. Who maintains this: a low-code team or engineers? How complex is the flow really? And can you review it a year from now? For the deeper mechanics of building agentic workflows in Logic Apps, I covered that ground in the Logic Apps Agent Loop series so that I won’t repeat it here.
Where API Management fits
Azure API Management (APIM) isn’t messaging or orchestration. Instead, it’s the control point in front of both.
It sits between consumers and whatever does the real work: a Logic App, a Function, an App Service backend. From there, it enforces rate limits, authentication, transformation, and policy-based routing. So when multiple consumers hit a shared set of backend capabilities, APIM lets you change the implementation behind them without breaking anyone, and lets you enforce policy without touching application code.
That’s all I’ll say here, because APIM earns a series of its own. I went deep on it in the APIM for AI workloads series, including how it behaves as an AI gateway. For this layer, treat it as the front door that governs whatever messaging and orchestration sit behind it.
Where each is the wrong answer
Every service here has a failure mode when you reach for it by reflex. So, to keep this honest:
Service Bus is wrong for high-volume telemetry: If you’re routing millions of fire-and-forget events and none of them individually matter, Service Bus is expensive overkill. Use Event Grid.
Event Grid is wrong for anything needing order: The moment sequence or guaranteed delivery matters, Event Grid stops fitting. Move to Service Bus before the gap bites.
Logic Apps is wrong past its complexity ceiling: A workflow with thirty actions and nested branches is a maintenance liability in the designer. Decompose it, or move it to code.
Code is wrong for something a citizen developer should own: Not every integration belongs in a repo. If a low-code team can own and maintain a simple connector-driven flow, hand-writing it in a Function just centralizes work that didn’t need to be centralized.
The shape of it
Azure gives you four integration services, but you don’t choose between four things. You answer two questions. Is this messaging or orchestration? And then what does failure cost, and who owns the work? Answer those, and Service Bus, Event Grid, Logic Apps, or a Function each falls out naturally, with APIM governing the front.
In practice, real platforms use several together: APIM, fronting Logic Apps, and Functions, Event Grid fanning out to Service Bus for reliable processing. So the craft of Azure messaging and orchestration isn’t picking a winner. It’s drawing clean boundaries between them.
Want the layer above this one? The Azure PaaS map puts integration in context against compute, data, and governance, and walks the five-question framework for choosing across all of them.
In the previous post we added conversation persistence to the Microsoft Foundry Citadel Platform on Azure. As a result, every agent run now produces a structured document in the spoke’s Cosmos DB conversations container. The agent is fully operational: it routes through the APIM governance hub, executes tool calls, stores its history, and returns grounded responses. However, the question that every enterprise AI architect eventually faces remains: what happens when it needs to stop?
Not a graceful shutdown. Not a redeployment. An immediate, operator-triggered containment the kind you need when an agent is behaving unexpectedly, consuming runaway tokens, or has been flagged by your security team. In a Microsoft Foundry Citadel Platform on Azure deployment, the answer is the Kill Switch: a layered containment system built into the APIM hub that stops agent traffic cold without touching the agent code, the spoke, or the Azure OpenAI deployment.
This post implements three of the five Citadel kill switch layers against the hub we deployed in Sweden Central:
Layer 1 — Named Value flip: instant global block via a single boolean
Layer 3 — Agent ID blocklist: surgical per-agent blocking
The Scenario
The weather agent (agent_with_memory.py) is running in production. Specifically, it is routing through apim-wpvlimv4ngkns.azure-api.net, storing conversations in the spoke Cosmos DB, and generating token usage events in the hub Cosmos DB. Everything is working. Then your security team flags it. The agent needs to stop immediately while the incident is investigated. You have seconds, not minutes.For example, redeploying the spoke takes too long. Rotating the APIM subscription key is irreversible and affects all consumers. Therefore, the Kill Switch is the right tool.
The Kill Switch is the right tool. The APIM hub has built-in pre-wiring, requires no code changes, and can trigger actions in under 30 seconds.
To ensure reliability, always pre-wire the kill switch as Layer 1 before you need it. Remember, you can’t flip a Named Value that doesn’t exist. In addition, the inbound policy must already be in place, checking the Named Value on every request, before any incident occurs.
Prerequisites
From the previous posts you should have:
Hub deployed in rg-ai-hub-gateway-dev with APIM instance apim-wpvlimv4ngkns
agent_with_memory.py running and saving to Cosmos DB
Azure CLI authenticated
Citadel Kill Switch Layer 1 — Named Value Flip
How It Works
A Named Value called kill-switch-enabled is created in APIM and set to false. An inbound policy on the OpenAI API checks this value on every request. When the value is flipped to true, all requests through the gateway immediately return HTTP 403 — no code changes, no redeployment, no spoke involvement.
Step 1.1 — Create the Named Value
azapimnvcreate`
--resource-grouprg-ai-hub-gateway-dev`
--service-nameapim-wpvlimv4ngkns`
--named-value-idkill-switch-enabled`
--display-name"kill-switch-enabled"`
--value"false"`
--secretfalse
Verify it was created:
azapimnvshow`
--resource-grouprg-ai-hub-gateway-dev`
--service-nameapim-wpvlimv4ngkns`
--named-value-idkill-switch-enabled`
--query"value"-otsv
Should return false.
Step 1.2 — Add the Inbound Policy
In the Azure Portal:
Navigate to apim-wpvlimv4ngkns → APIs → Azure OpenAI Service API → All operations
Click Policies → Inbound processing → Edit
Add this policy inside the <inbound> section, before any other policies:
<set-body>{"error": {"code": "KillSwitchActive", "message": "Agent access has been suspended by the governance hub. Contact your administrator.", "layer": 1}}</set-body>
</return-response>
</when>
</choose>
Click Save.
Step 1.3 — Confirm Agent Runs Normally
With kill-switch-enabled set to false, the agent should still work:
pythonagent_with_memory.py
Expected output: normal run, conversation saved, answer returned.
Step 1.4 — Trigger the Kill Switch
azapimnvupdate`
--resource-grouprg-ai-hub-gateway-dev`
--service-nameapim-wpvlimv4ngkns`
--named-value-idkill-switch-enabled`
--value"true"
Now run the agent:
pythonagent_with_memory.py
Expected output:
The agent stops. No spoke changes occur. No code changes happen. One CLI command executes.
The agent is required to pass a custom header x-agent-token containing a signed JWT with a specific claim (agt-approved: true). The APIM inbound policy validates this claim. If the claim is absent or the token is invalid, the system blocks the request with a 401 status. This action simulates identity-based containment, revoking the agent’s token or invalidating its claim at the identity provider level.
Step 2.1 — Update the Agent to Send a Header
Add the x-agent-token header to agent_with_memory.py. In this demo, we simulate the token by using a simple header value. IIn a production environment, Entra ID issues a JWT.
Modify the AzureOpenAI client creation in agent_with_memory.py:
client=AzureOpenAI(
azure_endpoint=apim_base,
api_key=cfg["APIM_SUBSCRIPTION_KEY"],
api_version="2024-02-01",
default_headers={
"x-agent-id": "citadel-weather-agent-v1"
}
)
Step 2.2 — Add the JWT Claim Check Policy
In the portal, add this policy after the Layer 1 block in the inbound section:
<set-body>{"error": {"code": "AgentNotApproved", "message": "Agent identity could not be verified. Approval header missing or invalid.", "layer": 2}}</set-body>
</return-response>
</when>
</choose>
Step 2.3 — Trigger Layer 2
Remove the x-agent-approved header from the agent (or set it to false) and run:
pythonagent_with_memory.py
Expected output:
Note the response header x-kill-switch-layer: 2-agent-approval this indicates which containment layer fired and is critical for incident triage.
Pitfall: Policy Order Matters
Layer 1 must appear before Layer 2 in the policy document. APIM evaluates inbound policies top to bottom and stops at the first <return-response>. If Layer 2 appears before Layer 1, a globally suspended agent would return a 401 (identity error) instead of a 403 (suspended), obscuring the true containment reason in your incident log.
Citadel Kill Switch Layer 3 — Agent ID Blocklist in APIM
How It Works
A Named Value called blocked-agent-ids holds a comma-separated list of agent IDs. The inbound policy checks the x-agent-id header against this list. When agents match, the system blocks them with a 403 status code. Non-matching agents continue operating normally. This approach allows for surgical containment, stopping one specific agent while allowing all others to function.
Step 3.1 — Create the Blocklist Named Value
azapimnvcreate`
--resource-grouprg-ai-hub-gateway-dev`
--service-nameapim-wpvlimv4ngkns`
--named-value-idblocked-agent-ids`
--display-name"blocked-agent-ids"`
--value"none"`
--secretfalse
Start with an empty value — no agents blocked.
Step 3.2 — Add the Blocklist Policy
Add this policy after Layer 2 in the inbound section:
<set-body>{"error": {"code": "AgentBlocked", "message": "Agent has been added to the governance blocklist.", "layer": 3}}</set-body>
</return-response>
</when>
</choose>
Step 3.3 — Add the Agent to the Blocklist
azapimnvupdate`
--resource-grouprg-ai-hub-gateway-dev`
--service-nameapim-wpvlimv4ngkns`
--named-value-idblocked-agent-ids`
--value"citadel-weather-agent-v1"
Run the agent:
pythonagent_with_memory.py
Expected output:
Step 3.4 — Surgical Validation
The power of Layer 3 is specificity. If you had a second agent with a different x-agent-id say citadel-docs-agent-v1 it would pass through Layer 3 unaffected while citadel-weather-agent-v1 remains blocked. One agent stopped, all others running. This is the enterprise AI governance pattern: granular control without broad disruption.
Validating the Citadel Kill Switch in Application Insights
Rather than using the CLI — which has a 5–10 minute Log Analytics ingestion lag — go directly to Application Insights in the portal for immediate results:
Portal → appi-apim-wpvlimv4ngkns in rg-ai-hub-gateway-dev
Left sidebar → Logs
Paste and run this query:
requests
|wheretimestamp> ago(2h)
|where resultCode in("200","401","403")
| project timestamp, resultCode, duration, name
|orderbytimestampdesc
The results table tells the complete kill switch story in two columns — resultCode and duration:
Application Insights Logs query on the Citadel APIM hub showing the kill switch in action, 401 responses at under 1ms confirm Layer 2 (agent approval header) blocking requests at the gateway before any LLM call is made, contrasted with normal 200 responses taking 683ms–2010ms for a full Azure OpenAI round trip.
The duration contrast is the definitive proof that the kill switch works as designed. The 401s and 403s resolve in under 50ms, stopped cold at the APIM inbound policy before a single token is sent to Azure OpenAI. The 200s take 683ms–2010ms because they made the full round trip through the governance hub to Azure OpenAI and back.
Zero tokens consumed on blocked requests, zero cost, and zero Cosmos DB writes in the spoke. The agent is stopped at the perimeter.
For a sharper view that highlights exactly which kill switch layer fired on each blocked request, add the response header to the query. Unfortunately APIM response headers are not automatically projected into the requests table in Application Insights — but you can distinguish the layers by combining result code and timing:
requests
|wheretimestamp> ago(2h)
|where resultCode in("200","401","403")
| extend killSwitchLayer =case(
resultCode =="401","Layer 2 — agent approval",
resultCode =="403"and duration <10,"Layer 1 or 3 — gateway block",
Application Insights Logs query on the Citadel APIM hub showing the kill switch incident log — Layer 2 agent approval blocks resolving in under 1ms with zero LLM calls made, contrasted with normal governed runs completing in 683ms–2010ms. The killSwitchLayer column identifies exactly which containment layer fired on each request.
This gives you a readable incident log showing which containment layer was active at each point in time, directly useful for DORA incident post-mortem documentation and EU AI Act Article 17 risk management records.
The Complete Three-Layer Kill Switch Policy
Here is the complete inbound policy block containing all three layers, ready to paste into APIM:
<!-- Kill Switch Layer 1: Named Value flip -->
<set-variable name="killSwitchActive" value="@("{{kill-switch-enabled}}" == "true")" />
<choose>
<when condition="@((bool)context.Variables["killSwitchActive"])">
<return-response>
<set-status code="403" reason="Agent Suspended" />
<set-header name="Content-Type" exists-action="override">
<value>application/json</value>
</set-header>
<set-header name="x-kill-switch-layer" exists-action="override">
<value>1-named-value</value>
</set-header>
<set-body>{"error": {"code": "KillSwitchActive", "message": "Agent access has been suspended by the governance hub.", "layer": 1}}</set-body>
</return-response>
</when>
</choose>
<!-- Kill Switch Layer 2: Agent approval header -->
<choose>
<when condition="@(context.Request.Headers.GetValueOrDefault("x-agent-approved", "false") != "true")">
<return-response>
<set-status code="401" reason="Agent Not Approved" />
<set-header name="Content-Type" exists-action="override">
<value>application/json</value>
</set-header>
<set-header name="x-kill-switch-layer" exists-action="override">
<value>2-agent-approval</value>
</set-header>
<set-body>{"error": {"code": "AgentNotApproved", "message": "Agent identity could not be verified. Approval header missing or invalid.", "layer": 2}}</set-body>
</return-response>
</when>
</choose>
<!-- Kill Switch Layer 3: Agent ID blocklist -->
<set-variable name="agentId" value="@(context.Request.Headers.GetValueOrDefault("x-agent-id", ""))" />
<set-variable name="blockedIds" value="@("{{blocked-agent-ids}}")" />
<choose>
<when condition="@{
var agentId = (string)context.Variables["agentId"];
var blockedIds = (string)context.Variables["blockedIds"];
if (string.IsNullOrEmpty(agentId) || string.IsNullOrEmpty(blockedIds)) { return false; }
return blockedIds.Split(',').Any(id => id.Trim() == agentId.Trim());
}">
<return-response>
<set-status code="403" reason="Agent Blocked" />
<set-header name="Content-Type" exists-action="override">
<value>application/json</value>
</set-header>
<set-header name="x-kill-switch-layer" exists-action="override">
<value>3-agent-blocklist</value>
</set-header>
<set-body>{"error": {"code": "AgentBlocked", "message": "Agent has been added to the governance blocklist.", "layer": 3}}</set-body>
</return-response>
</when>
</choose>
Pitfalls Summary
Pitfall
Fix
Named Value doesn’t exist at incident time
Pre-wire Layer 1 during normal operations — never during an incident
Policy evaluation error on {{kill-switch-enabled}}
Named Value must exist before the policy referencing it is saved
Layer 2 fires before Layer 1 in policy
Policy order matters — Layer 1 must be first in the inbound block
Agent ID header not sent
Add x-agent-id to default_headers in AzureOpenAI client
Blocklist with trailing spaces blocks nothing
Use .Trim() in the policy C# expression when splitting
Kill switch left active after test
Always reset Named Values after testing — kill-switch-enabled=false, blocked-agent-ids=""
What the Kill Switch Demonstrates About Citadel
The three layers reveal something important about the Citadel architecture: governance lives in the hub, not the agent. The agent code has no knowledge of the kill switch. The spoke has no kill switch configuration. The Azure OpenAI deployment is untouched. All containment logic is in the APIM hub’s inbound policy — one place, centrally managed, instantly effective.
This is the enterprise AI control plane pattern in practice. When an incident occurs:
Layer 1 stops everything immediately while you triage
Layer 2 enforces identity verification once normal operations resume
Layer 3 surgically targets the offending agent while other agents continue
The x-kill-switch-layer response header ensures your incident log captures exactly which containment mechanism fired, giving you a clean audit trail for post-mortem analysis — directly relevant for DORA incident reporting and EU AI Act Article 17 risk management documentation.
What’s Next
The next post in this series takes the dev setup and hardens it for non-prod: networkIsolation=true, APIM Premium SKU, per-spoke subscription keys with independent quotas, and Azure Policy at the management group level. The kill switch policies we built here carry forward unchanged governance in the hub environment, which is environment-agnostic.
In the Azure PaaS map post, App Service got just one paragraph. I called it the default for synchronous REST APIs that front a backend system. That summary is right, but Azure App Service architecture hides far more than one paragraph can carry.
When you stand up an App Service in a regulated enterprise, the interesting decisions sit around the app, not inside it. How does traffic reach it? And how does it authenticate outbound? Furthermore, how does it scale? And how do you ship changes without downtime? So this post takes a deeper look. I’ll walk the request path from the user to the data tier and flag the decisions that matter specifically for integration work.
The full picture request path across the top, the App Service Plan at the center, and the surrounding tiers of CI/CD, security, data, networking, and monitoring that make an integration platform work.
Azure App Service architecture: the request path, front to back
Before a request touches your code, it passes through a chain of services. Moreover, each one is a design decision rather than a default.
Every hop from user to data tier is a design decision, not a default, and for a single-region regulated platform, Application Gateway alone often does the job.
First, DNS resolves the URL.Azure DNS points the hostname at whatever sits in front of App Service. That step is trivial, but it’s worth naming, because the next hop depends on it.
Next, a global entry point handles routing and Web Application Firewall (WAF). Here the first real choice appears. Azure Front Door gives you global routing, CDN-style caching, and a Web Application Firewall at the edge. Therefore, you’d pick it when you serve a distributed audience or want TLS termination close to the user. Application Gateway, by contrast, performs Layer 7 load balancing and WAF regionally within your VNet. So you’d reach for it when traffic stays regional, and you want the firewall inside your network boundary. Plenty of designs use both Front Door globally, Application Gateway behind it. For a single-region platform in a regulated environment, though, Application Gateway alone often does the job. Better still, it keeps everything inside the VNet where your security team wants it.
Finally, App Service receives the request. By now, the traffic has been routed, load-balanced, and WAF-filtered. What App Service adds is a managed platform that runs your code, patches the underlying OS, and handles TLS for you.
Azure App Service architecture: Inside the App Service Plan
The App Service Plan catches people out. After all, this is where the platform allocates and bills are computed, not the app itself. Multiple apps can share one plan, so they also share its CPU and memory. That arrangement saves money until two apps contend under load. Then the “why is my API slow when the other app gets busy” investigation begins.
The plan is the billed compute unit. It load-balances instances across availability zones, and because multiple apps can share one plan, they also contend for its CPU and memory under load.
The plan defines a few things that matter architecturally:
Instances and scaling: The plan runs one or more instances, and App Service load-balances across them. Scale-out adds instances, which drives throughput. Scale-up swaps in bigger instances, which adds per-request headroom. Autoscale rules fire on metrics like CPU, memory, and HTTP queue length. That’s exactly why App Service suits steadily loaded request workloads rather than bursty, event-driven ones. So if your load is spiky and event-driven, consider Functions or Container Apps instead.
Availability zones: On tiers that support it, you can spread plan instances across zones. As a result, “highly available” ceases to be a claim and becomes an actual design property. For regulated production, treat this as table stakes rather than an upgrade.
Isolation: The isolated tiers run your plan in a dedicated environment inside your VNet, away from shared infrastructure. Therefore, you’d reach for them when compliance demands network isolation that shared tiers can’t provide a common requirement in health and finance.
The platform features integration architects actually use
App Service ships built-in capabilities that often do more work than the application code. Three of them earn their keep in every integration design:
Deploy and validate a new version in staging against production config, then swap it in instantly with an equally instant swap-back if something breaks.
Deployment slots are the single most useful feature for shipping without downtime. A staging slot lets you deploy, warm up, and validate a new version against production configuration. Then you swap it into production instantly, and swap back just as fast if something breaks. For a platform where a bad deploy takes down downstream consumers, that swap turns a potential incident into a controlled release.
Managed identity is the one I’d insist on. App Service can carry a system-assigned or user-assigned identity. Consequently, it authenticates to Key Vault, SQL, Service Bus, and Storage without a single connection string in the configuration. My first post made the same point about the governance layer. App Service is where you implement it for the compute tier.
VNet integration and private endpoints close the network. VNet integration lets the app call into your private network. Private endpoints let consumers reach the app privately, without a public address. In a regulated environment, you’ll usually want both. That way, the app communicates with backends over private links, and consumers access it through the gateway rather than a public URL.
The tiers around it: data, identity, observability
App Service never runs alone. In fact, the architecture around it is where an integration platform lives or dies:
Data services repeat the choices from the first post. Pick Azure SQL for relational integrity, Cosmos DB for flexible scaling, Blob Storage for files, and Redis Cache to absorb read load. App Service connects to all of them over private endpoints and authenticates through managed identity.
Security and identity means Entra ID for authentication, Key Vault for the secrets that can’t be an identity, and managed identities threading through everything. App Service’s built-in “Easy Auth” can offload the whole OIDC flow to the platform. That helps for internal APIs. Still, understand it before you lean on it for anything with complex authorization logic.
Monitoring and observability mean Application Insights and Azure Monitor. For an integration platform, this isn’t optional. When a request fails somewhere across the gateway, app, backend, and data tier, distributed tracing shows you where. So wire it in on day one, not after the first production incident.
Where App Service is the wrong answer
Let me keep this honest: the same point I make in every post. App Service isn’t always right, and reaching for it by reflex causes as many problems as it solves.
Does your workload run event-driven and bursty? Then App Service’s metric-based autoscale will lag the load or leave you overprovisioned. Functions or Container Apps fit better. Do you need long-running orchestration? App Service will host it, but you’re building a workflow engine on top of a request-serving platform. Logic Apps or Durable Functions exist for exactly that. Do you have real Kubernetes-native requirements? App Service won’t stretch that far, so that’s an AKS conversation.
App Service shines for steadily-loaded, request-driven APIs and backends. It gives you managed availability, easy TLS, slot-based deployment, and clean managed identity auth to the rest of the platform, for an integration platform that describes a large share of the synchronous surface. That’s exactly why it earns its place as the default compute tier as long as you know when to reach past it.
Azure App Service architecture: The shape of it
For an integration architect, Azure App Service architecture is mostly about what surrounds the app. Put a gateway with WAF in front, with private connectivity to backends and data. Use managed identity everywhere. Add slots for safe deployment. Trace the whole path. Get those right, and the app in the middle becomes almost boring, which, for a production platform, is the highest compliment there is.
Want the layer above this one? The Azure PaaS map puts App Service in context against Functions, Container Apps, and AKS. It also walks the five-question framework for choosing between them.
In the previous post, we connected a real tool-calling agent to the Microsoft Foundry Citadel Platform on Azure, routing every LLM call through the APIM governance hub in Sweden Central. The agent answered weather questions; the hub captured usage events in Cosmos DB; and Application Insights confirmed that both LLM calls were governed. The agent worked, but it had no memory. Every run started fresh, with no record of what was asked or answered.
This post adds conversation persistence to the Microsoft Foundry Citadel Platform on Azure. Every agent run now produces a structured document in the spoke’s Cosmos DB conversations container: the user’s question, the tool call made, the tool result, the agent’s answer, token counts, model version, and timestamp. The agent gains a memory layer, marking the transition as the spoke’s data tier becomes active.
What We Build
Each agent run writes one document to the spoke Cosmos DB:
{
"id":"run-20260625-143022-stockholm",
"principal_id":"steefjan@msn.com",
"timestamp":"2026-06-25T14:30:22.441Z",
"question":"What is the weather like in Stockholm right now?",
"tool_calls":[
{
"name":"get_weather",
"arguments":{"location":"Stockholm"},
"result":{
"location":"Stockholm, Sweden",
"temperature_celsius":22.5,
"wind_speed_kmh":7.2,
"condition":"Overcast"
}
}
],
"answer":"The weather in Stockholm is overcast with a temperature of 22.5°C...",
"model":"gpt-4o-2024-11-20",
"prompt_tokens":234,
"completion_tokens":67,
"total_tokens":301,
"apim_gateway":"apim-wpvlimv4ngkns.azure-api.net"
}
The partition key is /principal_id matching the container definition deployed by the spoke Bicep template. In addition, this arrangement ensures that all conversations for a given user are grouped into the same logical partition, making per-user history queries efficient.
The agent writes the document after completing the run, so a failed or incomplete run leaves no record.Moreover, it’s clean, simple, and auditable.
Prerequisites
From the previous two posts you should have:
Hub deployed in rg-ai-hub-gateway-dev
Spoke deployed in rg-ai-spoke-dev with Cosmos DB cosmos-tggi2gmkw22w4, database cosmos-dbtggi2gmkw22w4, container conversations
App Config appcs-tggi2gmkw22w4 populated with COSMOS_DB_ENDPOINT and CONVERSATIONS_DATABASE_CONTAINER
agent.py, config.py, and tools.py from the previous post
Virtual environment activated with openai, azure-appconfiguration, azure-identity, and requests installed
Step 1 — Install the Cosmos DB SDK
With your virtual environment activated:
pip install azure-cosmos
Pitfall: Cosmos DB Public Network Access
If your Cosmos DB has firewall rules enabled (which the spoke Bicep template sets by default), your local IP needs to be in the allowed list or public access needs to be set to All networks for dev. Check via the portal: cosmos-tggi2gmkw22w4 (your instance) → Networking → Public access → All networks → Save. In production this would be networkIsolation=true with private endpoints only.
Step 2 — Extend Config to Read Cosmos DB Settings
The spoke App Config already contains COSMOS_DB_ENDPOINT and CONVERSATIONS_DATABASE_CONTAINER — populated automatically during deployment. Extend config.py to pull these:
You should now see seven keys including COSMOS_DB_ENDPOINT pointing to https://cosmos-tggi2gmkw22w4.documents.azure.com:443/ and CONVERSATIONS_DATABASE_CONTAINER set to conversations.
query = f\"SELECT TOP {limit} c.id, c.timestamp, c.question, c.answer, c.total_tokens FROM c WHERE c.principal_id = @principal_id ORDER BY c._ts DESC\"
The CosmosClient with DefaultAzureCredential uses your Azure CLI identity locally. That identity needs the Cosmos DB Built-in Data Contributor role on the Cosmos DB account — not a standard Azure RBAC role, but a Cosmos DB data plane role. The spoke deployment should have assigned this automatically via the assignCosmosDBCosmosDbBuiltInDataContributorExecutor deployment. If you get a 403, verify:
az cosmosdb sql role assignment list `
--account-name cosmos-tggi2gmkw22w4 `
--resource-group rg-ai-spoke-dev `
--output table
Your principal ID (8e856fa1-f4c4-4a02-91a5-a6ccc6afc6b3) should appear with role definition ID ending in 00000000-0000-0000-0000-000000000002 (Built-in Data Contributor). If not, assign it:
Never use Cosmos DB connection strings or account keys in the agent code. The pattern here uses DefaultAzureCredential throughout — locally it picks up your az login identity, in production it uses the spoke’s Managed Identity. This is the NEN 7510 and cVGZ security baseline compliant approach.
Looking at a stored conversation document, every field serves a purpose:
Field
Purpose
id
Unique run identifier — traceable back to a specific agent invocation
principal_id
Partition key — enables per-user history queries and RBAC scoping
timestamp
ISO 8601 UTC — audit trail, correlatable with APIM logs
question
Original user input — searchable for pattern analysis
tool_calls
Full tool call log including arguments and results — debugging and audit
answer
Final agent response — quality review and feedback loops
model
Model version — tracks which model version answered which questions
prompt_tokens / completion_tokens
Cumulative across both LLM calls — accurate per-conversation cost
total_tokens
Sum of both calls — FinOps input per user per conversation
apim_gateway
Gateway used — identifies which hub instance served the request
The token counts here are cumulative across both LLM calls (tool decision and synthesis), yielding a true per-conversation costrather than a per-call figure. This is more useful for FinOps reporting you care about the cost of answering a question, not the cost of individual API calls within that answer.
Pitfalls Summary
Pitfall
Fix
Cosmos DB firewall blocks local IP
Portal → Networking → All networks for dev, or add specific IP
403 on Cosmos DB write
Assign Cosmos DB Built-in Data Contributor data plane role to your principal
CosmosResourceNotFoundError
Verify database name (cosmos-dbtggi2gmkw22w4) and container name (conversations) match exactly
Partition key mismatch
Container was created with /principal_id — every document must include this field
DefaultAzureCredential fails locally
Run az login and ensure the correct subscription is selected
Never use connection strings
Use DefaultAzureCredential throughout — locally via az login, in production via Managed Identity
What the Full Citadel Data Layer Now Looks Like
After this post, the spoke’s data tier is fully active:
The hub’s ai-usage-container captures the infrastructure view of every API call, governed and logged. The spoke’s conversations container captures the application view of every user interaction, structured and queryable. Together, they give you both compliance evidence and application telemetry from a single agent run.
What’s Next
The next post in this series showcases the Citadel Kill Switch and explains how it stops a governed agent when necessary. It details how the five-layer containment system in APIM effectively shuts down the process without affecting the spoke or agent code. The conversation history you’ve created illustrates the clear before-and-after contrast: requests flow to Cosmos DB and then abruptly halt at the gateway layer.
If you’ve spent any time in the Azure portal lately, you’ll know the problem isn’t a lack of PaaS services; it’s too many of them, with overlapping capabilities and just enough marketing gloss to make every option look like the right one. App Service or Container Apps? Logic Apps or Functions? Service Bus or Event Grid? The Azure PaaS catalog has grown quickly, and for integration architects specifically, the decisions compound: pick the wrong option at the compute layer, and you’re fighting the platform every time you add a connector, a retry policy, or a compliance control.
This post is a map, not a comparison matrix. I’m grouping the Azure PaaS services that matter to integration work into four layers: compute, integration, data, and governance, and walking through the decision points that arise when designing for a regulated enterprise environment rather than a greenfield demo. If you’ve followed my Logic Apps Agent Loop series or the APIM for AI workloads series, this sits underneath both the platform primer and the deeper posts, which assume you already have read.
Why Azure PaaS still matters to integration architects
IaaS provides you with a VM and asks you to manage everything above it. SaaS gives you the finished product and asks for nothing. PaaS sits in between: the platform owns patching, scaling, and availability, and you own the application logic and configuration. For integration workloads specifically, that trade-off is usually the right one: you rarely need to control the OS of a message broker, but you do need fine-grained control over routing, transformation, and policy enforcement.
The practical test I use: if a service requires you to think about instance sizing, OS patch cycles, or cluster upgrades, it’s leaning IaaS regardless of what the marketing page calls it. If it requires you to think about triggers, bindings, connectors, and scaling rules, it’s PaaS. AKS sits deliberately on that boundary; more on that below.
The four core PaaS layers for integration work compute, integration, data, and governance/identity plus the fifth layer, agentic workloads, expose: per-action authorization, compensating actions, and evaluation.
Layer 1: Compute in Azure PaaS Integration
Azure App Service
Still, the default is web APIs and backend services that don’t need event-driven scaling. App Service gives you deployment slots, built-in autoscale, and managed TLS with minimal ceremony. For integration architects, the main use case is hosting synchronous REST APIs that front a backend system the kind of thing that used to be a WCF service or an on-prem IIS site.
The limitation that catches people out: App Service scales on CPU/memory/queue-length rules, not on arbitrary event volume. If your workload is bursty and event-driven rather than steadily loaded, you’ll either overprovision or look elsewhere.
Azure Functions
The event-driven counterpart. Functions are the right choice when the unit of work is a discrete event: a message landing in a queue, a file arriving in Blob Storage, or an HTTP call that needs to fan out. The Consumption plan gives true scale-to-zero, which matters for cost in low-traffic integration scenarios; the Premium and Flex Consumption plans trade some of that elasticity for warm instances and VNet integration, which most enterprise integration platforms need anyway because you’re rarely allowed to expose a public endpoint without a private link in front of it.
Where Functions get uncomfortable: long-running orchestrations. A single function execution has a timeout, and while Durable Functions solves the orchestration problem, you’re now managing a stateful workflow engine on top of a stateless compute primitive. That’s usually the point where I ask whether Logic Apps would do the job with less code.
Azure Container Apps
The newer entrant is increasingly my default recommendation for anything that needs to run a container without the operational overhead of Kubernetes. Container Apps gives you KEDA-based event-driven scaling, Dapr integration for service-to-service calls and pub/sub, and revision-based traffic splitting all without you touching a node pool. For integration architects building agent-based or microservice-style integration components, this is often the sweet spot: you get container portability (useful if the workload might move, or if you’re standardizing on containers for other reasons) without inheriting cluster lifecycle management.
Azure Kubernetes Service (AKS)
Worth naming even though it’s not strictly Azure PaaS: Microsoft manages the control plane, yet you still own node pool upgrades, networking configuration, and workload scheduling. AKS earns its place when you have genuine Kubernetes-native requirements: custom operators, a multi-team platform where Kubernetes is the common substrate, or workloads that need capabilities Container Apps doesn’t expose yet. For most integration teams, reaching for AKS by default is over-engineering. Reach for it when a specific requirement forces your hand, not because it’s the more “serious” option.
Layer 2: Integration with Azure PaaS
Azure Logic Apps
The workflow orchestration layer with Azure PaaS remains the most direct route to enterprise connectors: SAP, IBM MQ, mainframe hosts, and the long tail of line-of-business systems that lack a modern REST API. Standard Logic Apps (running on the single-tenant model) close most of the gaps that made Consumption Logic Apps hard to use in regulated environments: VNet integration, built-in state management, and per-workflow scaling.
The honest trade-off: Logic Apps designer-first workflows are fast to build and easy for less code-heavy teams to maintain, but they get harder to reason about and harder to code-review once a workflow grows past a certain complexity. I’ve found the practical ceiling is somewhere around “a dozen actions with a couple of branches.” Past that, either decompose into smaller workflows or move the logic into a Function.
Azure API Management
Not just a gateway for integration architects, APIM is where governance actually gets enforced. Rate limiting, authentication, request/response transformation, and policy-based routing all live here, in front of whatever compute layer is doing the real work. If you’re building any platform where multiple consumers hit a shared set of backend capabilities, APIM is the control point that lets you change backend implementations without breaking consumers and enforce policy without touching application code.
The thing worth planning for early: policy authoring in APIM is a distinct skill, separate from the languages your team already knows. Please budget time for the team to learn the policy XML dialect rather than treating it as an afterthought. Badly written policies are a common source of latency and hard-to-diagnose failures.
Azure Service Bus
The durable, ordered, transactional messaging backbone. Reach for Service Bus when you need guaranteed delivery, sessions for ordered processing, or transactional message handling across multiple operations. Topics and subscriptions give you pub/sub without standing up a separate broker.
Azure Event Grid
The lightweight, high-throughput event router. Where Service Bus is about reliable delivery of business messages, Event Grid is about routing high-volume, fire-and-forget events resource state changes, custom application events, IoT telemetry to whichever subscriber cares about them. The two are frequently used together: Event Grid fans out a notification, and a subscriber puts a durable message on Service Bus for guaranteed processing.
A rule of thumb I use with teams new to Azure integration: if losing a message would be a business incident, it belongs on Service Bus. If losing a message would just mean a missed notification, Event Grid is fine.
Layer 3: Data in Azure PaaS
Integration architecture lives and dies by what’s underneath it, and the PaaS data services matter as much as the compute and messaging layers.
Azure SQL Database remains the default for relational, transactional workloads with a need for strong consistency think reference data, transactional state, anything with real foreign-key relationships. In addition, Azure Cosmos DB earns its place when you need global distribution, flexible schema, or the kind of horizontal scale that a single SQL instance won’t give you cheaply; it’s also increasingly the default choice for conversation and state storage in agentic workloads, given its low-latency reads and flexible document model. Finally, Azure Cache for Redis sits in front of both, absorbing read load and giving you a fast, ephemeral store for session state or short-lived coordination data.
The mistake I see most often: teams default to Cosmos DB because it’s the “modern” choice, then discover they actually needed relational integrity and end up hand-rolling consistency checks that SQL would have given them for free. Pick based on the access pattern, not the reputation.
Layer 4: Governance and identity for Azure PaaS
This is the layer that separates a proof of concept from something you can run in a regulated industry. Microsoft Entra ID and managed identities remove the need for connection strings and API keys scattered across configuration files. Each of the Azure PaaS services above should authenticate via managed identity. Key Vault holds what can’t be a managed identity (third-party API keys, certificates).Azure Policy and Microsoft Defender for Cloud provide the guardrails and posture visibility that an auditor or security team will ask for. Azure Monitor and Application Insights are non-negotiable for integration platforms, especially when a message fails somewhere in a chain of five services. Distributed tracing is the difference between a five-minute diagnosis and a day of log archaeology.
Layer 5: The gap that agentic workloads expose
Everything above holds for conventional integration platforms. Agentic AI workloads add a wrinkle that a recent round of discussion on LinkedIn I saw around an enterprise agent architecture diagram put well: the model is arguably the least differentiated part of a production agent deployment. Identity, permissions, observability, governance, and reliable orchestration are what separate a working demo from something you can run against real systems, and a few of those deserve to be called out specifically for integration architects, because they don’t map cleanly onto the governance layer above.
The identity perimeter validates the caller once, at the edge. A poisoned tool result enters the reasoning loop from the data the agent requested and never crosses that boundary. Per-action authorization is the control that reaches inside the loop where the threat actually lives.
Idenitity
Identity secures who the agent is, not what it does. Entra ID and managed identity answer “Is this caller who it claims to be?” They don’t address what happens when a poisoned tool result or a manipulated retrieved document changes the agent’s next action mid-reasoning loop. Prompt injection rides in through the RAG layer and tool outputs inside the loop, where identity checks at the perimeter don’t reach. The practical implication for PaaS design is that authorization needs to occur per action, not just per identity. This is exactly what APIM policy scoping and per-tool consent in Logic Apps and AI Foundry connectors are for: to treat each tool call as its own authorization decision, rather than an inherited privilege from a validated caller.
Recovery
Recovery means compensating actions, not just retries. Agent actions have side effects across systems: a ticket got created, a record got updated, an email went out. A failed step three actions into an agent loop can’t just retry from the top; it needs a saga-style compensating action to undo what already happened. Service Bus sessions and Logic Apps’ native support for scoped try/catch-with-compensation are the building blocks here — but the compensation logic has to be designed in explicitly, because neither service provides it by default.
Evaluation
Evaluation and drift detection are a first-class layer, not an afterthought. Application Insights and Azure Monitor provide operational observability into latency, error rates, and throughput. They don’t tell you if the agent’s outputs are quietly degrading in quality over time. That’s a separate concern, and one worth budgeting for from the start rather than bolting on after the first bad production incident.
The questions worth asking before an agent goes anywhere near a real system: what did it access, what tool did it call, why did it act, what policy constrained it, what happened when it failed, and who owns the outcome. If a PaaS architecture can’t answer all six, the gap isn’t in the model; it’s in the platform around it.
Azure PaaS Integration decisions: a framework, not a decision tree for integration architects
None of these layers are picked in isolation; the compute choice constrains the integration pattern, and the integration pattern constrains what the data layer needs to support. When I’m working through this with a team, the questions I ask in order are:
Is the trigger an event or a schedule/request? Event-driven points toward Functions or Container Apps with KEDA; request-driven points toward App Service or APIM-fronted compute.
Does a human or a low-code team need to maintain this workflow? If yes, Logic Apps earns serious consideration even if a Function would be more “elegant.”
What’s the cost of losing a message? Business-critical → Service Bus. Best-effort notification → Event Grid.
Does the data need strong relational integrity, or flexible scale? SQL for the former, Cosmos DB for the latter — and don’t let the “modern” label make the decision for you.
Is everything behind managed identity and traced end to end? If the answer is no anywhere in the chain, that’s the next thing to fix, not the last.
The five questions as a branching flow trigger type, workflow ownership, message-loss cost, data access pattern, and the managed-identity gate that comes before anything ships.
Azure PaaS Integration Conclusion
That’s the shape of it. In practice, most enterprise integration platforms end up using several of these services together: API Management fronting a mix of Logic Apps and Functions, backed by Service Bus for reliable delivery and Cosmos DB or SQL for state and the architecture work is less about picking a single winner than about drawing clean boundaries between them.
In the previous post we deployed a working Microsoft Foundry Citadel Platform on Azure Sweden Central, a Governance Hub built on Azure API Management and an Agent Spoke built on Azure AI Foundry. We validated the setup with a raw chat completion call through the APIM gateway. That proved the plumbing works. This post takes the next step: connecting a real tool-calling agent to the Microsoft Foundry Citadel Platform on Azure, using the Open-Meteo weather API as a tool, and showing that every LLM call flows through the hub’s governance layer.
The agent is built with the standard Azure OpenAI SDK pointed directly at the Citadel APIM gateway. It uses a custom function tool that calls the Open-Meteo API to retrieve real current weather data for any location. The governance hub intercepts all traffic: content safety policies fire, token usage is tracked, and telemetry flows into Application Insights. This is the Microsoft Foundry Citadel Platform doing what it is designed to do.
What We Build
The flow looks like this:
End-to-end flow of a tool-calling agent on the Microsoft Foundry Citadel Platform in Azure Sweden Central, the Python agent routes both LLM calls through the APIM Governance Hub, executes the get_weather tool against Open-Meteo, and receives a grounded response, with all traffic captured in Application Insights and Cosmos DB.
Two LLM calls flow through APIM per agent run: the tool decision call and the synthesis call. Both are governed, appear in Application Insights, and contribute to Cosmos DB usage tracking.
Why Open-Meteo and Why the Standard OpenAI SDK
The original plan was to use the Azure AI Foundry Agent Service SDK with Bing Search grounding. Two blockers emerged:
Bing Search SKU eligibility: The Grounding with Bing Search resource (G1 SKU) requires Pay-As-You-Go or EA subscriptions and is not available on MVP or MSDN subscriptions.
AI Foundry Agent Service routing: The azure-ai-projects SDK routes LLM calls through the AI Foundry project’s internal endpoint (aif-tggi2gmkw22w4.openai.azure.com) rather than through APIM, bypassing the governance layer. In addition, even after adding APIM as a connected resource in the AI Foundry portal, the Agent Service does not honor it for model routing in the current preview version.
The solution, therefore, is to use the standard OpenAI Python SDK pointed directly at the APIM gateway endpoint. This guarantees that all traffic flows through the hub; consequently, the tool-calling loop is implemented explicitly in Python, and the governance telemetry is fully captured in Application Insights.
Open-Meteo is a free, open-source weather API; therefore, it requires no API key and returns structured JSON weather data. Additionally, it serves as a clean stand-in for any external API your agents might call in production.
Prerequisites
From the previous post you should have:
Hub deployed in rg-ai-hub-gateway-dev with APIM gateway URL https://apim-wpvlimv4ngkns.azure-api.net and subscription key
Spoke deployed in rg-ai-spoke-dev with App Config appcs-tggi2gmkw22w4 containing APIM_GATEWAY_URL and APIM_SUBSCRIPTION_KEY
Your principal ID with App Configuration Data Reader role on the spoke App Config
For this post you additionally need Python 3.11 or later installed locally.
Step 1 — Set Up the Python Environment
mkdir citadel-agent && cd citadel-agent
python -m venv .venv
# Windows
.venv\Scripts\activate
pip install openai
pip install azure-appconfiguration
pip install azure-identity
pip install requests
Step 2 — Read Configuration from App Config
Create config.py using Set-Content to avoid BOM issues on Windows:
All four keys should return truncated values. If you get a 403, wait 2–5 minutes for role assignment propagation and retry.
Pitfall: Always Use WriteAllLines for Python Files on Windows
Out-File -Encoding utf8NoBOM and @"..."@ | Out-File both add a BOM on some Windows PowerShell versions, causing Python to throw SyntaxError: Non-UTF-8 code starting with '\xff'. Use [System.IO.File]::WriteAllLines with [System.Text.UTF8Encoding]::new($false) to write files without BOM.
The AzureOpenAI SDK constructs the full path as {azure_endpoint}/openai/deployments/{model}/chat/completions. If your APIM_GATEWAY_URL in App Config contains /openai at the end, strip it before passing to the client; otherwise, the SDK builds a doubled path (/openai/openai/...) that returns a 500 from APIM. The line apim_base = cfg['APIM_GATEWAY_URL'].rstrip('/').replace('/openai', '') handles this automatically.
After running the agent, check Application Insights in the hub:
az monitor app-insights query `
--app <YourAPIMinstanceName> `
--resource-group rg-ai-hub-gateway-dev `
--analytics-query "requests | where timestamp > ago(10m) | project timestamp, name, resultCode, duration | order by timestamp desc" `
--output table
Pitfall: CLI vs Portal Ingestion Lag
The CLI query hits the Log Analytics store; however, it has a 5–10-minute ingestion lag. In contrast, the Azure Portal Application Insights blade uses a live metrics path and shows results immediately. Therefore, if the CLI returns an empty response, it’s a good idea to check the portal directly, go to the APIM instance → Performance to view requests in real time.
What Governed Traffic Looks Like in the Portal
The Application Insights Performance blade shows two operation types per agent run:
azure-openai-service-api:rev=1 - ChatCompletions_Create — the APIM policy-matched operation, showing the governed calls with content safety applied
POST /openai/openai/deployments/chat/chat/completions — the raw endpoint calls
Each agent run generates two successful requests (tool decision + synthesis), both with response code 200 and latency around 900ms–1.2s for gpt-4o. Failed attempts from earlier endpoint format issues show as 500s and are clearly distinguishable.
Application Insights Performance blade for the Citadel Governance Hub, confirming agent traffic routed through APIM: 9 requests captured, with the governed ChatCompletions_Create operation averaging 1.09 seconds, and all successful calls returning response code 200.
The Azure AI Foundry Agent Service SDK — What We Learned
For completeness, here is a summary of what we discovered when attempting to use the azure-ai-projects SDK before switching to the standard OpenAI SDK:
Issue
Detail
FunctionTool import path
Must import from azure.ai.agents.models, not azure.ai.projects.models
create_thread does not exist
Use create_thread_and_process_run instead
list_messages does not exist
Use client.agents.messages.list(thread_id=...)
MessageRole.ASSISTANT does not exist
Use the string "assistant" directly
enable_auto_function_calls(toolset=...) fails
Parameter is tools=, not toolset=
Function not found error
Call client.agents.enable_auto_function_calls(tools=toolset) before create_agent
Agent traffic bypasses APIM
AI Foundry Agent Service uses its own endpoint resolution — use standard OpenAI SDK pointed at APIM instead
The Agent Service SDK is in active beta development (azure-ai-agents==1.2.0b6 at the time of writing). Expect these APIs to stabilise and the APIM routing issue to be addressed in future versions.
Pitfalls Summary
Pitfall
Fix
Grounding with Bing Search G1 SKU not eligible
Requires Pay-As-You-Go or EA subscription
Bing.Search.v7 CLI creation fails
Resource type moved to Microsoft.Bing/accounts
BOM in Python files on Windows
Use [System.IO.File]::WriteAllLines with UTF8Encoding($false)
APIM endpoint doubles /openai path
Strip /openai from URL before passing to AzureOpenAI client
App Config 403 on first run
Wait 2–5 minutes for role assignment propagation
CLI Application Insights query empty
5–10 minute ingestion lag — check portal Performance blade instead
AI Foundry Agent Service bypasses APIM
Use standard openai SDK pointed directly at APIM gateway
What the Full Citadel Loop Delivers
With the agent running through APIM, every LLM call in the tool-calling loop is governed:
Content Safety — both the user question and the synthesised response pass through Azure AI Content Safety policies configured in APIM.
Token tracking — each of the two LLM calls contributes to the token usage log in Cosmos DB, giving you per-call cost attribution by APIM subscription key. The Cosmos DB ai-usage-container in the hub captures a structured document for each LLM call, including the model version, token counts, gateway region, request IP, APIM subscription name, backend routing, and timestamp. In production, the productName field maps to the APIM subscription key. Aggregating documents by this field gives you direct FinOps reporting per AI initiative.
The Citadel hub Cosmos DB ai-usage-container showing a usage document captured from the tool-calling agent run model gpt-4o-2024-11-20, 70 total tokens, gateway region Sweden Central, routed via apim-wpvlimv4ngkns. Every LLM call through APIM generates a document like this, which serves as the cost attribution and audit trail for enterprise AI governance.
Latency observability — Application Insights captures the duration of every call, making it easy to identify slow tool calls or model latency spikes.
Audit trail — every request is logged with timestamp, operation name, response code, and duration. For a healthcare or financial services context, this is your compliance evidence.
What’s Next
This post wires a tool-calling agent to the Citadel hub using the standard OpenAI SDK. The natural next steps:
Azure AI Foundry Agent Service routing — as the SDK matures, the azure-ai-projects client will likely gain proper APIM gateway support. Watch the azure-ai-agents release notes for updates on connection-based routing.
Conversation persistence — store conversation history in the Cosmos DB conversations container already deployed in the spoke. The App Config key CONVERSATIONS_DATABASE_CONTAINER points to it.
Network isolation — re-enable networkIsolation=true in the spoke parameters to route all traffic through private endpoints.
Multiple tools — extend the agent with additional function tools (document lookup, product catalog, claims system) using the same pattern. Each tool call flows through APIM and is governed identically.
Conclusion
Connecting a real tool-calling agent to the Microsoft Foundry Citadel Platform on Azure requires three components: the standard OpenAI SDK configured to point to the APIM gateway, a function tool with a JSON schema definition, and an explicit tool-call-handling loop. Everything else, governance, content safety, token tracking, and cost attribution, is handled by the Citadel hub automatically.
The path to get here involved navigating several SDK beta rough edges and discovering that the AI Foundry Agent Service bypasses APIM in its current preview form. These are expected friction points with a platform in active development. The governance architecture underneath is sound, the APIM policies work, and the Application Insights telemetry confirms it.
Two LLM calls. Both governed. Both visible. That is what the Citadel hub delivers.
