Cosmos DB Agent Memory Cost: Caching, RU Drivers, and a Pitfalls Roundup

Posted on September 17, 2026 by steefjan1970

Post 6 of 6 on Cosmos DB agent memory cost: the hard numbers post 1 promised back at the start of this series.

Post 1 opened with a 2017 CloudBrew reviewer calling a Cosmos DB proof of concept “an hour-long marketing pitch,” and my answer then was that cost means nothing without the revenue it enables. Five posts later, that argument still needs the numbers behind it.

Where This Post Picks Up

This post closes the series with them: what actually drives Cosmos DB agent memory cost at the RU level, how semantic caching cuts LLM spend specifically, what to monitor once an agent runs in production, and a pitfalls roundup that pulls every thread from posts 2 through 5 into one list.

Semantic Caching: Reusing What You Already Paid to Compute

An LLM call is almost always the most expensive, highest-latency step in an agent’s request path, far more than any Cosmos DB read or write. A semantic cache cuts that cost by skipping the LLM entirely when a close-enough answer already exists. Instead of matching prompts by exact string, it vectorizes the incoming prompt and runs a similarity search against the prompt-completion pairs already sitting in the cache. The mechanics are the same as the VectorDistance() query from post 3; only the container changes, from memory to cache.

Two details make this different from a normal cache, and both matter for cost control. First, the similarity threshold is a real trade-off, not a default to leave alone: set it too high, and near-identical questions still miss and hit the LLM anyway; set it too low, and the cache starts returning answers that don’t actually match what the user meant. Second, a semantic cache needs the same context window as an LLM.

Cache only the raw prompt, and two different users who each ask “what’s the second largest?” in unrelated conversations get whichever answer the cache stored first, correct for one thread, wrong for the other. Vectorize a slice of the conversation history alongside the latest prompt, the way post 2’s turn-based schema already structures it, and the cache lookup carries the same context the LLM would have used. TTL handles cleanup the same way it does for turns in post 2, with one addition worth considering: a hit-count field that increments on each cache hit lets a pruning pass keep frequently reused entries around longer than questions the cache only ever answered once.

What Actually Drives Cosmos DB Agent Memory Cost

Four decisions drive most of the RU bill for an agent workload, and they’re not evenly weighted. Partition skew usually costs the most: at Cosmos DB Conf 2026, an engineer described a production account running at 100% RU utilization, throttling and retrying under load, where the obvious fix looked like provisioning more throughput. The real cause turned out to be a single logical partition absorbing over 80% of traffic, one automated integration account driving most writes under a partition key that looked reasonable on paper. Fixing the data model, without adding a single RU of throughput, dropped utilization to 20–35% and made the throttling disappear entirely. More throughput would have masked that problem, not fixed it.

Item size and shape matter next, and this series already covered the mechanism in post 2: one document per turn keeps writes small and cheap. At the same time, one-document-per-thread turns every new message into a full-item rewrite that gets steadily more expensive as the thread grows. Vector index choice is the third lever. DiskANN’s sharding and approximate search solve a scale problem post 3 already flagged, and paying for that complexity below roughly ten thousand vectors buys nothing quantizedFlat wasn’t already providing. TTL is the fourth: expired short-term memory that never actually expires, because someone set a container-level default once and never came back to it, quietly inflates storage and index size on data nobody queries anymore.

The Fifth Lever: Consolidation

There’s a fifth lever underneath all four, and it’s the one this whole series has been arguing for since post 1: consolidation. Running a cache, a relational store, and a dedicated vector database as three separate systems means paying for three separate throughput allocations, three separate operational surfaces, and cross-system network cost on every request that touches more than one of them. One Cosmos DB account carrying memory, search, and cache together shares throughput across all three instead of over-provisioning each in isolation; the 2017 CloudBrew critique missed the same argument when it judged the account’s line-item cost without asking what running three systems instead of one would have cost by comparison.

Monitoring: What to Watch Once It’s Running

Three signals catch most problems before they become an incident. Change feed lag matters most for the multi-agent handoffs post 4 covered — a growing lag between a write and the Function that reacts to it means a specialist agent is falling behind the conversation, not just running a little slower. Break RU consumption out per container instead of watching one account-wide total, and it shows which specific workload is driving spend — turns, checkpoints, or the semantic cache — instead of leaving that as a guess. And for catching expensive patterns before they ship at all, the Azure Cosmos DB VS Code extension’s Query Insights and Index Advisor flag cross-partition queries, missing filters, and indexing gaps directly in the editor, well before a query shape becomes production traffic.

Pitfalls Roundup: Every Thread from This Series

Unsharded vector index in a multitenant app (post 3) — without a vectorIndexShardKey, semantic search scans every tenant’s vectors, not just the current one.
Thread-per-item growth (post 2) — an item that grows by one append per turn gets more expensive to write with every message, and eventually hits a hard size limit.
Missing or forgotten TTL (post 2) — short-term memory nobody set an expiration for keeps sitting in the container indefinitely, quietly inflating storage.
Cross-tenant memory leakage (posts 3 and 4) — a global vector index or an unscoped checkpoint container lets one tenant’s context bleed into another’s.
Treating change feed as globally ordered (post 4) — ordering holds within a partition key, never across the whole container.
Confusing Foundry Agent Service Classic and New containers (post 5) — the newest trap in the list, and already the most common source of “why is my thread storage empty” reports.

What I’d Ask the Product Team

Multi-region writes for a globally distributed agent multiply throughput cost by the number of regions. The guidance so far is “add regions only where traffic justifies it,” which is reasonable. Still, it leaves the actual crossover point (how much traffic, at what latency requirement) for each team to work out through trial and error rather than a documented formula. A cost calculator that takes a workload shape and a target latency and outputs a recommended region count would save a lot of that guesswork.

Where This Is the Wrong Answer

Not every agent workload belongs on one account. A workload with one enormous, narrowly specialized vector search needs tens of billions of vectors; nothing else can still get better unit economics from a dedicated vector database that specializes in exactly that shape, rather than a general-purpose store carrying memory, search, and cache together. The unified argument holds for the vast majority of agent workloads this series has covered, not for every workload unconditionally.

Closing the Series

That 2017 reviewer wasn’t wrong that the account cost more than a bare-minimum alternative; the miss was judging that cost without the workload it made possible, the same mistake the Figma AWS costs piece argued against in a completely different context. Six posts and one real production case study later, Cosmos DB agent memory cost comes down to the same handful of decisions this series has covered since post 2: partition key, item shape, index choice, and TTL, with semantic caching and consolidation compounding the savings on top. That’s the whole series in one sentence, and it’s the argument I’d have made at CloudBrew in 2017 if I’d had the RU numbers to back it up yet.

Sources

Microsoft Learn — Semantic cache for large language models
Azure Cosmos DB Blog — 7 tips to optimize Azure Cosmos DB costs for AI and agentic workloads.
Microsoft for Developers — Azure Cosmos DB Conf 2026 Recap: Lessons from Production
Your own archive — Figma AWS Costs Explained

Cosmos DB Foundry Agent Service: Bring-Your-Own Thread Storage

Posted on September 16, 2026 by steefjan1970

Post 5 of 6 on Cosmos DB Foundry Agent Service integration, owning the thread store instead of leaving it opaque behind a managed API.

Posts 1 through 4 assumed you manage the Cosmos DB account directly. Foundry Agent Service changes that assumption by default: spin up an agent the basic way, and Microsoft manages the thread store for you, out of reach of a direct query. Standard agent setup flips that around. Cosmos DB Foundry Agent Service integration lets threads, system messages, and agent metadata land in a Cosmos DB account you own, sitting right where the schema, search, and checkpointing patterns from the rest of this series already apply.

Why Bring-Your-Own Thread Storage Matters

Data residency, security review, and auditability all get harder when a vendor holds conversation history in a store you can’t query. Standard setup solves that by provisioning three customer-owned resources instead of one managed black box: Azure Storage for uploaded files, Azure AI Search for the agent’s vector stores, and Azure Cosmos DB for everything: conversational messages, threads, and agent metadata. Cosmos DB carries the load that matters most for this series: it’s where the actual conversation lives.

Inside enterprise_memory: Cosmos DB Foundry Agent Service Containers

Standard setup names the resulting database enterprise_memory, and container names inside it depend entirely on which Foundry Agent Service runtime the agent runs on. Foundry Agent Service (Classic) writes to three containers: thread-message-store for end-user conversation messages, system-thread-message-store for internal system messages, and agent-entity-store for agent metadata like instructions and tools. Foundry Agent Service (New) writes to two different containers instead of agent-definitions-v1 and run-state-v1, and neither runtime reads the other’s containers. Check thread-message-store for an agent running on the New runtime, and it comes back empty, not because BYO thread storage failed, but because the data landed somewhere else entirely.

Provisioning Standard Agent Resources

Provisioning means more than a Cosmos DB account on its own. Standard setup also expects an Azure Storage account, an Azure AI Search resource, and an Azure Key Vault for secrets, alongside a deployed agent-compatible model. Once those exist, Microsoft’s Bicep template accepts the resource IDs of existing accounts and wires up the rest: account and project connections, role assignments, and the capability hosts that tell Foundry where agent state actually lives

Two things catch people off guard here, so it’s worth flagging both before you deploy. First, throughput: your Cosmos DB account needs at least 3,000 RU/s total 1,000 RU/s for each of the three baseline containers and that number scales up with every additional project sharing the account, since each project gets its own container set. Undershoot it, and the deployment doesn’t fail quietly; it throws CapabilityHostProvisioningFailed during the capability host step. Second, roles: the project’s managed identity needs Cosmos DB Operator at the account level to provision containers, plus Cosmos DB Built-in Data Contributor at the database level for enterprise_memory. The database-level scope covers every container inside it, so a single role assignment handles the whole set instead of one per container.

Querying Thread History Directly

This is what bring-your-own thread storage actually buys over the default: a direct line into conversation history that Foundry’s own API doesn’t expose. Once a thread exists in thread-message-store (Classic) or run-state-v1 (New), you can run the same vector, full-text, and hybrid queries from post 3 against it — same RANK RRF(...) syntax, same partition-scoped WHERE clause. The only difference is the container: Foundry manages it instead of your own code.

Treat this as a connection into Foundry Agent Service, though, not a replacement for it. Foundry still owns thread creation, run orchestration, and tool invocation; Cosmos DB Foundry Agent Service integration only changes where the resulting data sits and who can query it directly.

Pitfalls

Checking the wrong container set. The single most common source of “BYO thread storage isn’t working” reports is querying Classic’s containers for an agent running on the New runtime, or vice versa. Confirm which runtime a project uses before assuming a missing thread means a broken connection.

Underprovisioning throughput for multiple projects. The 3,000 RU/s floor covers one project’s container set. Add a second project to the same Cosmos DB account, and the container count and the RU/s requirement under it doubles. CapabilityHostProvisioningFailed almost always traces back to this, not to a misconfigured connection.

Assuming you can edit a capability host after creation. You can’t. Pointing a project capability host at the wrong Cosmos DB resource ID means deleting and recreating the project, not patching the connection — worth getting right on the first deployment rather than treating it as a setting to adjust later.

Next: Cost, Caching, and Production Pitfalls

That settles Cosmos DB Foundry Agent Service integration for readers who need full ownership of thread storage rather than a managed default. The last post in this series pulls back to the practitioner-notes view: RU cost drivers, semantic caching, and a closing pitfalls roundup across everything posts 2 through 5 have covered.

Sources

Microsoft Learn — Integration with Foundry Agent Service
Microsoft Learn — Set up standard agent resources for Foundry Agent Service.
GitHub — Standard agent setup Bicep template
Azure Cosmos DB Blog — Azure AI Foundry Connection for Azure Cosmos DB and BYO Thread Storage

Designing a Cosmos DB Agent Memory Schema

Posted on September 1, 2026 by steefjan1970

Post 2 of 6 on Cosmos DB agent memory schema design before you write a line of agent code.

Post 1 made the case for one database instead of three. This post is about the decisions that determine whether that one database actually holds up: partition key and item shape, both of which you choose before an agent ever writes a turn. Get these wrong, and no amount of DiskANN or 99.999% SLA saves you from a hot partition or a rewrite-the-whole-item cost curve later.

Partition Keys: The Foundation of a Cosmos DB Agent Memory Schema

Because Cosmos DB automatically partitions data, the partition key is the single most consequential choice in the schema. It decides how writes distribute, how queries scope, and eventually how much a bad decision costs to unwind. Three strategies cover most agent memory scenarios, and each trades distribution for locality differently.

Partition Keys

GUID as the partition key. Every item lands in its own logical partition. Writes distribute as evenly as possible, so this works well for high-volume, write-heavy logging where you rarely need to reassemble a conversation later think raw telemetry more than chat history. The cost shows up on read: reconstructing a thread means a cross-partition query.
threadId as the partition key. All turns in a conversation share one partition key, so “give me the last 10 turns” or “vector search within this thread” both stay inside a single partition. This is the default for conversational agents and RAG apps, provided threads are numerous and varied enough to avoid concentrating writes on a few hot values.
[tenantId, threadId] as a hierarchical partition key. This is where the series comes full circle: I covered hierarchical partition keys in Azure Cosmos DB’s Latest Performance Features back in 2023, for the same tenant-then-item pattern. Nothing about the mechanism changed for agent workloads: threads still colocate under their tenant, tenant-level queries still avoid scanning every partition, and Cosmos DB still sub-partitions past the 20 GB logical-partition ceiling the same way it always did. Only the data moving through it is new.

There’s no universally correct answer here, so match the strategy to the query pattern that matters most: threadId if “give me this conversation” dominates, GUID if raw write throughput dominates, hierarchical if tenant isolation is a governance requirement and not just a nice-to-have.

Three Ways to Shape the Memory Item

Partition key decides where data lives. Item shape decides what it costs to read and write once it’s there, and that’s the other half of a Cosmos DB agent memory schema.

One document per turn (recommended default). Each item holds a complete exchange: a user prompt, the agent’s reply, and any tool call in between, and carries threadId, turnIndex, and an embedding alongside it. Most single- and multi-agent apps default to this shape, because it balances a small, cheap-to-write item against enough context to be useful on its own. “Latest N turns” is a simple ORDER BY turnIndex query, and ttl can expire old turns individually instead of touching the whole thread.

One document per response. Every user message, agent reply, and tool result gets its own item, all sharing a threadId. This is the most granular option, useful when you need to embed and search every single utterance independently, but it multiplies item count and RU cost on read. It loses the natural “question and answer together” unit that a semantic cache wants.

One document per thread. The whole conversation lives in one item that grows with every append. Reading the full history is a single read, which sounds appealing until a long-running thread turns every new turn into a full-item rewrite. Treat this as an anti-pattern unless the thread is short and you can bound its length by design: a five-turn onboarding flow, maybe; an open-ended assistant conversation, no.

TTL as Memory Lifecycle Management

Short-term memory should disappear on its own, and time-to-live is how Cosmos DB does that without a cleanup job. Set a default ttl on the container, and every item expires after that many seconds unless it overrides the value itself; set the container default to -1 instead, and Cosmos DB turns on ttl without expiring anything, unless an item sets its own positive ttl field. That second mode is the more useful one for agent memory, since it lets long-term memories (ttl: -1 on the item, meaning never expire) sit in the same container as short-term turns (ttl: 3600, gone in an hour) without a separate container or a background job doing the deleting.

A Worked Schema

Here’s the turn-based model from the previous section as an actual container and item.

Create the container with a hierarchical partition key, and turn on ttl at the container level:

			
az cosmosdb sql container create \
  --account-name my-cosmos-account \
  --database-name agentmemory \
  --name turns \
  --resource-group my-rg \
  --partition-key-path "/tenantId" "/threadId" \
  --ttl -1

		

A single turn, ready to insert:

			
{
  "id": "b9c5b6ce-2d9a-4a2b-9d76-0f5f9b2a9a91",
  "tenantId": "contoso",
  "threadId": "thread-1234",
  "turnIndex": 7,
  "messages": [
    { "role": "user", "content": "What's our refund policy for accessories?" },
    { "role": "agent", "content": "Refund policy is 30 days for unopened items." }
  ],
  "embedding": [0.013, -0.092, 0.551],
  "ttl": 3600
}

		

And writing it with the Python SDK:

			
from azure. cosmos import CosmosClient
client = CosmosClient(url, credential)
container = client.get_database_client("agentmemory").get_container_client("turns")
container.upsert_item({
    "id": "b9c5b6ce-2d9a-4a2b-9d76-0f5f9b2a9a91",
    "tenantId": "contoso",
    "threadId": "thread-1234",
    "turnIndex": 7,
    "messages": [
        {"role": "user", "content": "What's our refund policy for accessories?"},
        {"role": "agent", "content": "Refund policy is 30 days for unopened items."},
    ],
    "embedding": [0.013, -0.092, 0.551],
    "ttl": 3600,
})

		

Swap ttl: 3600 for ttl: -1 on any item you want to keep past the container default as summarized long-term memory; for instance, it survives while the rest of the thread ages out on schedule.

Seeing It Live

The rest of this section shows the same schema running against a real Cosmos DB account, not just described on paper.

The provision.py creating the turns container with the [/tenantId, /threadId] hierarchical partition key and container-level ttl.

The same container in Data Explorer’s Scale & Settings pane partition key and ttl exactly as provision.py set them.

A seeded turn in Data Explorer’s Items view tenantId, threadId, turnIndex, messages, and embedding, matching the JSON above field for field.

Pitfalls

Hot partitions from low-cardinality tenant keys. A hierarchical [tenantId, threadId] key only distributes well if tenants themselves are numerous and reasonably balanced in volume. One enterprise customer generating 80% of total traffic under a single tenantId value creates a hot partition no amount of threadId variety underneath it fixes. Check tenant volume distribution before committing to this key, not after.

Unbounded thread-per-item growth. It’s tempting to reach for one-document-per-thread because “just read the whole conversation” feels simpler in application code. In practice, an item that grows by one append per turn racks up RU cost on every single write as the item gets larger, and Cosmos DB item size limits eventually cap how long a thread can run at all. If a thread’s length isn’t predictable and short, don’t model it this way.

Forgetting embeddings need to be top-level fields. If you nest an embedding array inside a messages object, it won’t be indexable for vector search — it has to sit at the top level of the item, alongside threadId and turnIndex, for the container’s vector policy to pick it up. This one is easy to miss because the item still writes successfully; it just never shows up in a vector query, and that failure is silent until someone notices recall is worse than expected.

Next: Finding the Right Memory

The schema in this post gets data into Cosmos DB efficiently. It doesn’t yet get the right memory back out at the moment an agent needs it — that’s the job of vector, full-text, and hybrid search, which is where post 3 picks up. That settles the Cosmos DB agent memory schema; retrieval is the next problem worth solving properly.

Sources

Microsoft Learn — Agent memories in Azure Cosmos DB for NoSQL
Microsoft Learn — Time to live (TTL) in Azure Cosmos DB.
Microsoft Learn — Hierarchical partition keys
Your own archive — Azure Cosmos DB’s Latest Performance Features
GitHub — steefjan1/cosmos-agent-memory-lab — the runnable sample behind this schema, and behind posts 3 and 4

Why Cosmos DB Ends Up as the Agent Memory Database

Posted on August 30, 2026 by steefjan1970

The first post in a series on Cosmos DB agent memory for AI agents, starting nine years before “AI agent” was a category.

In 2017, I built a proof of concept for a customer: a knowledge base on Cosmos DB, using the Graph model and Search, running at roughly 1,000 euros a month. I presented it at CloudBrew. One attendee wasn’t impressed:

“The most uninteresting talk of the day came from Steef-Jan Wiggers, who, in my opinion, delivered an hour-long marketing pitch for CosmosDB. I think it’s expensive for what it currently offers, and many developers could architect something with just as much performance without needing CosmosDB.”

He wasn’t wrong that 1,000 euros a month raises eyebrows as a line item. He was wrong about what the line item paid for: the knowledge base was the product a subscription business planned to sell. Compare the cost to the revenue it enabled, and it’s negligible. Compare it to nothing, and of course it looks “expensive.” I made the same point about Figma’s AWS bill last year: $109 million a year sounds alarming until you check it against $821 million in revenue and a business model that requires sub-100ms real-time collaboration for 13 million users. In short, cost without context is just a number that sounds big.

So here’s the same argument, nine years later, with a different workload. Agent memory, chat turns, tool call results, embeddings, and user preferences are expensive to store the wrong way and reasonably cheap to store the right way, and increasingly “the right way” means one database instead of three. I’ll come back to the actual RU numbers in post 6; for now, this post is about why the architecture argument holds up before cost even enters the picture.

The Same Shape of Problem, Nine Years Apart

Strip away “AI agent” and look at what you’re actually storing: short-lived, high-volume, time-ordered records that need fast writes and selective recall. That’s chat turns and tool outputs today. It’s also, structurally, what I modeled in a Cosmos DB Conf 2023 talk on end-to-end retail process monitoring, messages and batches flowing between an ERP, a WMS, and a PIM system, which I tracked so a retailer could tell where something broke.

Different domain, same shape, though: append-heavy writes, a need to reconstruct “what happened, in order,” and a downstream system (an incident manager then, an LLM now) that needs the right slice of history on demand, not the whole history every time.

In general, agent memory falls into two categories:

Short-term (episodic/working) memory — the last 5–10 turns of a conversation, intermediate tool call results, partial task state. Useful for the current task, disposable afterward (Cosmos DB’s time-to-live feature is a natural fit here; more on that in post 2).
Long-term memory — user preferences, summarized threads, facts the agent should persist and recall across sessions.

Both need somewhere to live, but the default answer for the last few years has been: somewhere different.

Why the Stitched Stack Breaks Down for Cosmos DB Agent Memory

The common pattern from 2022 through 2025 was to give each concern its own database: an in-memory store for caching and session state, a relational database for operational data and conversation logs, a purpose-built vector database for embeddings. A reasonable instinct, in theory: each tool for its own job.

In practice, though, it doesn’t hold up once an agent is the thing reading and writing across all three, on every turn.

Each piece has a real weakness once agents are the workload, not an afterthought:

Pure vector databases tend to offer no strong read/write guarantees, limited ingestion throughput, availability below 99.9%, a single (eventual) consistency level, and thin multitenancy support. Fine for an embeddings side-project. Shaky as the record of what an agent told a customer.
Relational databases fight the fluid, evolving schema of agent state, new fields, new memory types, and nested tool outputs without migrations and, often, downtime.
In-memory caches are fast and don’t persist, which is exactly the opposite of what long-term memory needs.

As a result, three systems also means three consistency models, three availability profiles, and three places a multi-agent system can silently desynchronize. And that complexity tax doesn’t show up in any single service’s bill, which is part of why it’s easy to miss until something breaks in production.

The Unified Case — and How Much of It I’d Already Used

The pitch for Cosmos DB agent memory as a unified layer rests on a small set of properties: single-digit-millisecond latency, a 99.999% availability SLA on the NoSQL API, DiskANN-based vector indexing built into the same store as the operational data, multi-master writes, and five selectable consistency levels from strong to eventual. In plain terms, that’s one system that’s fast enough for the hot path, available enough for production, and flexible enough to hold embeddings next to the record they came from.

In fact, two of the pieces that make this work aren’t new to me, or new to this blog.

Change feed, for example. In the retail monitoring solution, change feed was the mechanism that turned a write into a trigger: a new record landing in Cosmos DB fired a Function, which could raise an incident. That’s the same primitive I’ll use in post 4 to coordinate handoffs between agents in a multi-agent system: one agent’s write becomes another agent’s signal to act, without polling.

Hierarchical partition keys, likewise. I covered these in Azure Cosmos DB’s Latest Performance Features back in 2023: partitioning by tenant, then by item, to keep related data colocated while avoiding the 20 GB logical partition ceiling. The mechanism hasn’t changed; what’s changed is the workload. Post 2 uses the same [tenantId, threadId] pattern to isolate one customer’s agent conversations from another’s.

Even so, I didn’t build either feature for AI agents. Both turned out to be exactly what agent memory needs a decent sign that the underlying database was solid before the AI use case arrived, and nobody retrofitted it to fit.

Where This Series Is Headed

This post is the framing argument for Cosmos DB agent memory. From here, the rest of the series gets specific:

Post 2 — designing the agent memory schema itself: partition key choice, TTL, and the turn-based data model that works best in practice.
Post 3 — vector, full-text, and hybrid search for recalling the right memories, not just any memories.
Post 4 — multi-agent state and coordination, including change feed as the handoff mechanism.
Post 5 — wiring Cosmos DB into Microsoft Foundry Agent Service as bring-your-own thread storage.
Post 6 — the cost conversation, properly this time: RU drivers, semantic caching, and what this actually costs to run at scale.

The 2017 knowledge base cost 1,000 euros a month and paid for itself many times over as a revenue-generating product. Ultimately, the question worth asking about agent memory infrastructure in 2026 isn’t “is this expensive”; it’s the same question it always was: expensive relative to what?

Sources

Microsoft Learn — AI agents in Azure Cosmos DB
Microsoft Learn — Time to live (TTL) in Azure Cosmos DB.
Your own archive — Azure Cosmos DB’s Latest Performance Features
Your own archive — Figma AWS Costs Explained: Beyond the Hype and Panic
Microsoft Learn Shows — Leveraging Azure Cosmos DB for End-to-End Monitoring of Retail Processes

Microsoft Foundry Citadel Platform Azure: Connecting a Tool-Calling Agent

Posted on July 1, 2026 by steefjan1970

In the previous post we deployed a working Microsoft Foundry Citadel Platform on Azure Sweden Central, a Governance Hub built on Azure API Management and an Agent Spoke built on Azure AI Foundry. We validated the setup with a raw chat completion call through the APIM gateway. That proved the plumbing works. This post takes the next step: connecting a real tool-calling agent to the Microsoft Foundry Citadel Platform on Azure, using the Open-Meteo weather API as a tool, and showing that every LLM call flows through the hub’s governance layer.

The agent is built with the standard Azure OpenAI SDK pointed directly at the Citadel APIM gateway. It uses a custom function tool that calls the Open-Meteo API to retrieve real current weather data for any location. The governance hub intercepts all traffic: content safety policies fire, token usage is tracked, and telemetry flows into Application Insights. This is the Microsoft Foundry Citadel Platform doing what it is designed to do.

What We Build

The flow looks like this:

Two LLM calls flow through APIM per agent run: the tool decision call and the synthesis call. Both are governed, appear in Application Insights, and contribute to Cosmos DB usage tracking.

Why Open-Meteo and Why the Standard OpenAI SDK

The original plan was to use the Azure AI Foundry Agent Service SDK with Bing Search grounding. Two blockers emerged:

Bing Search SKU eligibility: The Grounding with Bing Search resource (G1 SKU) requires Pay-As-You-Go or EA subscriptions and is not available on MVP or MSDN subscriptions.

AI Foundry Agent Service routing: The azure-ai-projects SDK routes LLM calls through the AI Foundry project’s internal endpoint (aif-tggi2gmkw22w4.openai.azure.com) rather than through APIM, bypassing the governance layer. In addition, even after adding APIM as a connected resource in the AI Foundry portal, the Agent Service does not honor it for model routing in the current preview version.

The solution, therefore, is to use the standard OpenAI Python SDK pointed directly at the APIM gateway endpoint. This guarantees that all traffic flows through the hub; consequently, the tool-calling loop is implemented explicitly in Python, and the governance telemetry is fully captured in Application Insights.

Open-Meteo is a free, open-source weather API; therefore, it requires no API key and returns structured JSON weather data. Additionally, it serves as a clean stand-in for any external API your agents might call in production.

Prerequisites

From the previous post you should have:

Hub deployed in rg-ai-hub-gateway-dev with APIM gateway URL https://apim-wpvlimv4ngkns.azure-api.net and subscription key
Spoke deployed in rg-ai-spoke-dev with App Config appcs-tggi2gmkw22w4 containing APIM_GATEWAY_URL and APIM_SUBSCRIPTION_KEY
Your principal ID with App Configuration Data Reader role on the spoke App Config

For this post you additionally need Python 3.11 or later installed locally.

Step 1 — Set Up the Python Environment

			
mkdir citadel-agent && cd citadel-agent
python -m venv .venv
# Windows
.venv\Scripts\activate
pip install openai
pip install azure-appconfiguration
pip install azure-identity
pip install requests

		

Step 2 — Read Configuration from App Config

Create config.py using Set-Content to avoid BOM issues on Windows:

			
$lines = @(
    "from azure.appconfiguration import AzureAppConfigurationClient",
    "from azure.identity import DefaultAzureCredential",
    "",
    "APP_CONFIG_ENDPOINT = 'https://appcs-tggi2gmkw22w4.azconfig.io'",
    "LABEL = 'ai-lz'",
    "",
    "def get_config() -> dict:",
    "    credential = DefaultAzureCredential()",
    "    client = AzureAppConfigurationClient(",
    "        base_url=APP_CONFIG_ENDPOINT,",
    "        credential=credential",
    "    )",
    "    keys = [",
    "        'AI_FOUNDRY_PROJECT_ENDPOINT',",
    "        'CHAT_DEPLOYMENT_NAME',",
    "        'APIM_GATEWAY_URL',",
    "        'APIM_SUBSCRIPTION_KEY',",
    "    ]",
    "    config = {}",
    "    for key in keys:",
    "        setting = client.get_configuration_setting(key=key, label=LABEL)",
    "        config[key] = setting.value",
    "    return config",
    "",
    "if __name__ == '__main__':",
    "    cfg = get_config()",
    "    for k, v in cfg.items():",
    "        print(f'{k}: {v[:30]}...')"
)
[System.IO.File]::WriteAllLines("$PWD\config.py", $lines, [System.Text.UTF8Encoding]::new($false))

		

Test it:

python config.py

All four keys should return truncated values. If you get a 403, wait 2–5 minutes for role assignment propagation and retry.

Pitfall: Always Use WriteAllLines for Python Files on Windows

Out-File -Encoding utf8NoBOM and @"..."@ | Out-File both add a BOM on some Windows PowerShell versions, causing Python to throw SyntaxError: Non-UTF-8 code starting with '\xff'. Use [System.IO.File]::WriteAllLines with [System.Text.UTF8Encoding]::new($false) to write files without BOM.

Step 3 — Define the Weather Tool

Create tools.py:

			
$lines = @(
    "import json",
    "import requests",
    "",
    "def get_weather(location: str) -> str:",
    "    try:",
    "        geo = requests.get(",
    "            'https://geocoding-api.open-meteo.com/v1/search',",
    "            params={'name': location, 'count': 1, 'language': 'en', 'format': 'json'},",
    "            timeout=10",
    "        )",
    "        geo.raise_for_status()",
    "        geo_data = geo.json()",
    "        if not geo_data.get('results'):",
    "            return json.dumps({'error': f'Location not found: {location}'})",
    "        r = geo_data['results'][0]",
    "        weather = requests.get(",
    "            'https://api.open-meteo.com/v1/forecast',",
    "            params={'latitude': r['latitude'], 'longitude': r['longitude'], 'current_weather': True, 'wind_speed_unit': 'kmh', 'timezone': 'auto'},",
    "            timeout=10",
    "        )",
    "        weather.raise_for_status()",
    "        c = weather.json()['current_weather']",
    "        codes = {0:'Clear sky',1:'Mainly clear',2:'Partly cloudy',3:'Overcast',45:'Foggy',61:'Slight rain',63:'Moderate rain',65:'Heavy rain',71:'Slight snow',80:'Showers',95:'Thunderstorm'}",
    "        return json.dumps({'location': f'{r[chr(110)+(chr(97)+chr(109)+chr(101))]}, {r.get(chr(99)+chr(111)+chr(117)+chr(110)+chr(116)+chr(114)+chr(121),chr(32))}', 'temperature_celsius': c['temperature'], 'wind_speed_kmh': c['windspeed'], 'wind_direction_degrees': c['winddirection'], 'condition': codes.get(c['weathercode'],'Unknown'), 'is_day': bool(c['is_day'])})",
    "    except Exception as e:",
    "        return json.dumps({'error': str(e)})",
    "",
    "WEATHER_TOOL_DEFINITION = {",
    "    'type': 'function',",
    "    'function': {",
    "        'name': 'get_weather',",
    "        'description': 'Get current weather for a location. Returns temperature in Celsius, wind speed, condition.',",
    "        'parameters': {",
    "            'type': 'object',",
    "            'properties': {'location': {'type': 'string', 'description': 'City name e.g. Stockholm'}},",
    "            'required': ['location']",
    "        }",
    "    }",
    "}"
)
[System.IO.File]::WriteAllLines("$PWD\tools.py", $lines, [System.Text.UTF8Encoding]::new($false))

		

Test it:

python -c "from tools import get_weather; print(get_weather('Stockholm'))"

Step 4 — Create the Agent

Create agent.py using the standard openai SDK pointed directly at the APIM gateway:

			
$lines = @(
    "import json",
    "from openai import AzureOpenAI",
    "from config import get_config",
    "from tools import get_weather, WEATHER_TOOL_DEFINITION",
    "",
    "def run_agent(user_question: str) -> str:",
    "    cfg = get_config()",
    "",
    "    # Strip /openai suffix - AzureOpenAI SDK adds it automatically",
    "    apim_base = cfg['APIM_GATEWAY_URL'].rstrip('/').replace('/openai', '')",
    "",
    "    client = AzureOpenAI(",
    "        azure_endpoint=apim_base,",
    "        api_key=cfg['APIM_SUBSCRIPTION_KEY'],",
    "        api_version='2024-02-01',",
    "    )",
    "",
    "    messages = [{'role': 'user', 'content': user_question}]",
    "    print(f'Sending request via APIM: {apim_base}')",
    "",
    "    # First LLM call - agent decides whether to use the tool",
    "    response = client.chat.completions.create(",
    "        model=cfg['CHAT_DEPLOYMENT_NAME'],",
    "        messages=messages,",
    "        tools=[WEATHER_TOOL_DEFINITION],",
    "        tool_choice='auto',",
    "    )",
    "",
    "    msg = response.choices[0].message",
    "    messages.append(msg)",
    "",
    "    # Handle tool calls if the agent decided to use get_weather",
    "    if msg.tool_calls:",
    "        for tool_call in msg.tool_calls:",
    "            args = json.loads(tool_call.function.arguments)",
    "            print(f'  -> Tool call: get_weather({args})')",
    "            result = get_weather(**args)",
    "            print(f'  -> Tool result: {result}')",
    "            messages.append({",
    "                'role': 'tool',",
    "                'tool_call_id': tool_call.id,",
    "                'content': result,",
    "            })",
    "",
    "        # Second LLM call - synthesise grounded response",
    "        response = client.chat.completions.create(",
    "            model=cfg['CHAT_DEPLOYMENT_NAME'],",
    "            messages=messages,",
    "        )",
    "        return response.choices[0].message.content",
    "",
    "    return msg.content",
    "",
    "if __name__ == '__main__':",
    "    question = 'What is the weather like in Stockholm right now?'",
    "    print(f'Question: {question}')",
    "    answer = run_agent(question)",
    "    print(f'Answer: {answer}')"
)
[System.IO.File]::WriteAllLines("$PWD\agent.py", $lines, [System.Text.UTF8Encoding]::new($false))

		

Run it:

python agent.py

A successful run looks like this:

Pitfall: APIM Endpoint Format

The AzureOpenAI SDK constructs the full path as {azure_endpoint}/openai/deployments/{model}/chat/completions. If your APIM_GATEWAY_URL in App Config contains /openai at the end, strip it before passing to the client; otherwise, the SDK builds a doubled path (/openai/openai/...) that returns a 500 from APIM. The line apim_base = cfg['APIM_GATEWAY_URL'].rstrip('/').replace('/openai', '') handles this automatically.

After running the agent, check Application Insights in the hub:

			
az monitor app-insights query `
  --app <Your APIM instance Name> `
  --resource-group rg-ai-hub-gateway-dev `
  --analytics-query "requests | where timestamp > ago(10m) | project timestamp, name, resultCode, duration | order by timestamp desc" `
  --output table

		

Pitfall: CLI vs Portal Ingestion Lag

The CLI query hits the Log Analytics store; however, it has a 5–10-minute ingestion lag. In contrast, the Azure Portal Application Insights blade uses a live metrics path and shows results immediately. Therefore, if the CLI returns an empty response, it’s a good idea to check the portal directly, go to the APIM instance → Performance to view requests in real time.

What Governed Traffic Looks Like in the Portal

The Application Insights Performance blade shows two operation types per agent run:

azure-openai-service-api:rev=1 - ChatCompletions_Create — the APIM policy-matched operation, showing the governed calls with content safety applied
POST /openai/openai/deployments/chat/chat/completions — the raw endpoint calls

Each agent run generates two successful requests (tool decision + synthesis), both with response code 200 and latency around 900ms–1.2s for gpt-4o. Failed attempts from earlier endpoint format issues show as 500s and are clearly distinguishable.

The Azure AI Foundry Agent Service SDK — What We Learned

For completeness, here is a summary of what we discovered when attempting to use the azure-ai-projects SDK before switching to the standard OpenAI SDK:

Issue	Detail
`FunctionTool` import path	Must import from `azure.ai.agents.models`, not `azure.ai.projects.models`
`create_thread` does not exist	Use `create_thread_and_process_run` instead
`list_messages` does not exist	Use `client.agents.messages.list(thread_id=...)`
`MessageRole.ASSISTANT` does not exist	Use the string `"assistant"` directly
`enable_auto_function_calls(toolset=...)` fails	Parameter is `tools=`, not `toolset=`
Function not found error	Call `client.agents.enable_auto_function_calls(tools=toolset)` before `create_agent`
Agent traffic bypasses APIM	AI Foundry Agent Service uses its own endpoint resolution — use standard OpenAI SDK pointed at APIM instead

The Agent Service SDK is in active beta development (azure-ai-agents==1.2.0b6 at the time of writing). Expect these APIs to stabilise and the APIM routing issue to be addressed in future versions.

Pitfalls Summary

Pitfall	Fix
Grounding with Bing Search G1 SKU not eligible	Requires Pay-As-You-Go or EA subscription
`Bing.Search.v7` CLI creation fails	Resource type moved to `Microsoft.Bing/accounts`
BOM in Python files on Windows	Use `[System.IO.File]::WriteAllLines` with `UTF8Encoding($false)`
APIM endpoint doubles `/openai` path	Strip `/openai` from URL before passing to `AzureOpenAI` client
App Config 403 on first run	Wait 2–5 minutes for role assignment propagation
CLI Application Insights query empty	5–10 minute ingestion lag — check portal Performance blade instead
AI Foundry Agent Service bypasses APIM	Use standard `openai` SDK pointed directly at APIM gateway

What the Full Citadel Loop Delivers

With the agent running through APIM, every LLM call in the tool-calling loop is governed:

Content Safety — both the user question and the synthesised response pass through Azure AI Content Safety policies configured in APIM.

Token tracking — each of the two LLM calls contributes to the token usage log in Cosmos DB, giving you per-call cost attribution by APIM subscription key. The Cosmos DB ai-usage-container in the hub captures a structured document for each LLM call, including the model version, token counts, gateway region, request IP, APIM subscription name, backend routing, and timestamp. In production, the productName field maps to the APIM subscription key. Aggregating documents by this field gives you direct FinOps reporting per AI initiative.

Latency observability — Application Insights captures the duration of every call, making it easy to identify slow tool calls or model latency spikes.

Audit trail — every request is logged with timestamp, operation name, response code, and duration. For a healthcare or financial services context, this is your compliance evidence.

What’s Next

This post wires a tool-calling agent to the Citadel hub using the standard OpenAI SDK. The natural next steps:

Azure AI Foundry Agent Service routing — as the SDK matures, the azure-ai-projects client will likely gain proper APIM gateway support. Watch the azure-ai-agents release notes for updates on connection-based routing.

Conversation persistence — store conversation history in the Cosmos DB conversations container already deployed in the spoke. The App Config key CONVERSATIONS_DATABASE_CONTAINER points to it.

Network isolation — re-enable networkIsolation=true in the spoke parameters to route all traffic through private endpoints.

Multiple tools — extend the agent with additional function tools (document lookup, product catalog, claims system) using the same pattern. Each tool call flows through APIM and is governed identically.

Conclusion

Connecting a real tool-calling agent to the Microsoft Foundry Citadel Platform on Azure requires three components: the standard OpenAI SDK configured to point to the APIM gateway, a function tool with a JSON schema definition, and an explicit tool-call-handling loop. Everything else, governance, content safety, token tracking, and cost attribution, is handled by the Citadel hub automatically.

The path to get here involved navigating several SDK beta rough edges and discovering that the AI Foundry Agent Service bypasses APIM in its current preview form. These are expected friction points with a platform in active development. The governance architecture underneath is sound, the APIM policies work, and the Application Insights telemetry confirms it.

Two LLM calls. Both governed. Both visible. That is what the Citadel hub delivers.

Cloud Perspectives

Steef-Jan Wiggers

Tag Archives: AI Architecture

Cosmos DB Agent Memory Cost: Caching, RU Drivers, and a Pitfalls Roundup

Where This Post Picks Up

Semantic Caching: Reusing What You Already Paid to Compute

What Actually Drives Cosmos DB Agent Memory Cost

The Fifth Lever: Consolidation

Monitoring: What to Watch Once It’s Running

Pitfalls Roundup: Every Thread from This Series

What I’d Ask the Product Team

Where This Is the Wrong Answer

Closing the Series

Cosmos DB Foundry Agent Service: Bring-Your-Own Thread Storage

Why Bring-Your-Own Thread Storage Matters

Inside enterprise_memory: Cosmos DB Foundry Agent Service Containers

Provisioning Standard Agent Resources

Querying Thread History Directly

Pitfalls

Next: Cost, Caching, and Production Pitfalls

Designing a Cosmos DB Agent Memory Schema

Partition Keys: The Foundation of a Cosmos DB Agent Memory Schema

Partition Keys

Three Ways to Shape the Memory Item

TTL as Memory Lifecycle Management

A Worked Schema

Seeing It Live

Pitfalls

Next: Finding the Right Memory

Why Cosmos DB Ends Up as the Agent Memory Database

The Same Shape of Problem, Nine Years Apart

Why the Stitched Stack Breaks Down for Cosmos DB Agent Memory

The Unified Case — and How Much of It I’d Already Used

Where This Series Is Headed