Cosmos DB Agent Memory Cost: Caching, RU Drivers, and a Pitfalls Roundup

Posted on September 17, 2026 by steefjan1970

Post 6 of 6 on Cosmos DB agent memory cost: the hard numbers post 1 promised back at the start of this series.

Post 1 opened with a 2017 CloudBrew reviewer calling a Cosmos DB proof of concept “an hour-long marketing pitch,” and my answer then was that cost means nothing without the revenue it enables. Five posts later, that argument still needs the numbers behind it.

Where This Post Picks Up

This post closes the series with them: what actually drives Cosmos DB agent memory cost at the RU level, how semantic caching cuts LLM spend specifically, what to monitor once an agent runs in production, and a pitfalls roundup that pulls every thread from posts 2 through 5 into one list.

Semantic Caching: Reusing What You Already Paid to Compute

An LLM call is almost always the most expensive, highest-latency step in an agent’s request path, far more than any Cosmos DB read or write. A semantic cache cuts that cost by skipping the LLM entirely when a close-enough answer already exists. Instead of matching prompts by exact string, it vectorizes the incoming prompt and runs a similarity search against the prompt-completion pairs already sitting in the cache. The mechanics are the same as the VectorDistance() query from post 3; only the container changes, from memory to cache.

Two details make this different from a normal cache, and both matter for cost control. First, the similarity threshold is a real trade-off, not a default to leave alone: set it too high, and near-identical questions still miss and hit the LLM anyway; set it too low, and the cache starts returning answers that don’t actually match what the user meant. Second, a semantic cache needs the same context window as an LLM.

Cache only the raw prompt, and two different users who each ask “what’s the second largest?” in unrelated conversations get whichever answer the cache stored first, correct for one thread, wrong for the other. Vectorize a slice of the conversation history alongside the latest prompt, the way post 2’s turn-based schema already structures it, and the cache lookup carries the same context the LLM would have used. TTL handles cleanup the same way it does for turns in post 2, with one addition worth considering: a hit-count field that increments on each cache hit lets a pruning pass keep frequently reused entries around longer than questions the cache only ever answered once.

What Actually Drives Cosmos DB Agent Memory Cost

Four decisions drive most of the RU bill for an agent workload, and they’re not evenly weighted. Partition skew usually costs the most: at Cosmos DB Conf 2026, an engineer described a production account running at 100% RU utilization, throttling and retrying under load, where the obvious fix looked like provisioning more throughput. The real cause turned out to be a single logical partition absorbing over 80% of traffic, one automated integration account driving most writes under a partition key that looked reasonable on paper. Fixing the data model, without adding a single RU of throughput, dropped utilization to 20–35% and made the throttling disappear entirely. More throughput would have masked that problem, not fixed it.

Item size and shape matter next, and this series already covered the mechanism in post 2: one document per turn keeps writes small and cheap. At the same time, one-document-per-thread turns every new message into a full-item rewrite that gets steadily more expensive as the thread grows. Vector index choice is the third lever. DiskANN’s sharding and approximate search solve a scale problem post 3 already flagged, and paying for that complexity below roughly ten thousand vectors buys nothing quantizedFlat wasn’t already providing. TTL is the fourth: expired short-term memory that never actually expires, because someone set a container-level default once and never came back to it, quietly inflates storage and index size on data nobody queries anymore.

The Fifth Lever: Consolidation

There’s a fifth lever underneath all four, and it’s the one this whole series has been arguing for since post 1: consolidation. Running a cache, a relational store, and a dedicated vector database as three separate systems means paying for three separate throughput allocations, three separate operational surfaces, and cross-system network cost on every request that touches more than one of them. One Cosmos DB account carrying memory, search, and cache together shares throughput across all three instead of over-provisioning each in isolation; the 2017 CloudBrew critique missed the same argument when it judged the account’s line-item cost without asking what running three systems instead of one would have cost by comparison.

Monitoring: What to Watch Once It’s Running

Three signals catch most problems before they become an incident. Change feed lag matters most for the multi-agent handoffs post 4 covered — a growing lag between a write and the Function that reacts to it means a specialist agent is falling behind the conversation, not just running a little slower. Break RU consumption out per container instead of watching one account-wide total, and it shows which specific workload is driving spend — turns, checkpoints, or the semantic cache — instead of leaving that as a guess. And for catching expensive patterns before they ship at all, the Azure Cosmos DB VS Code extension’s Query Insights and Index Advisor flag cross-partition queries, missing filters, and indexing gaps directly in the editor, well before a query shape becomes production traffic.

Pitfalls Roundup: Every Thread from This Series

Unsharded vector index in a multitenant app (post 3) — without a vectorIndexShardKey, semantic search scans every tenant’s vectors, not just the current one.
Thread-per-item growth (post 2) — an item that grows by one append per turn gets more expensive to write with every message, and eventually hits a hard size limit.
Missing or forgotten TTL (post 2) — short-term memory nobody set an expiration for keeps sitting in the container indefinitely, quietly inflating storage.
Cross-tenant memory leakage (posts 3 and 4) — a global vector index or an unscoped checkpoint container lets one tenant’s context bleed into another’s.
Treating change feed as globally ordered (post 4) — ordering holds within a partition key, never across the whole container.
Confusing Foundry Agent Service Classic and New containers (post 5) — the newest trap in the list, and already the most common source of “why is my thread storage empty” reports.

What I’d Ask the Product Team

Multi-region writes for a globally distributed agent multiply throughput cost by the number of regions. The guidance so far is “add regions only where traffic justifies it,” which is reasonable. Still, it leaves the actual crossover point (how much traffic, at what latency requirement) for each team to work out through trial and error rather than a documented formula. A cost calculator that takes a workload shape and a target latency and outputs a recommended region count would save a lot of that guesswork.

Where This Is the Wrong Answer

Not every agent workload belongs on one account. A workload with one enormous, narrowly specialized vector search needs tens of billions of vectors; nothing else can still get better unit economics from a dedicated vector database that specializes in exactly that shape, rather than a general-purpose store carrying memory, search, and cache together. The unified argument holds for the vast majority of agent workloads this series has covered, not for every workload unconditionally.

Closing the Series

That 2017 reviewer wasn’t wrong that the account cost more than a bare-minimum alternative; the miss was judging that cost without the workload it made possible, the same mistake the Figma AWS costs piece argued against in a completely different context. Six posts and one real production case study later, Cosmos DB agent memory cost comes down to the same handful of decisions this series has covered since post 2: partition key, item shape, index choice, and TTL, with semantic caching and consolidation compounding the savings on top. That’s the whole series in one sentence, and it’s the argument I’d have made at CloudBrew in 2017 if I’d had the RU numbers to back it up yet.

Sources

Microsoft Learn — Semantic cache for large language models
Azure Cosmos DB Blog — 7 tips to optimize Azure Cosmos DB costs for AI and agentic workloads.
Microsoft for Developers — Azure Cosmos DB Conf 2026 Recap: Lessons from Production
Your own archive — Figma AWS Costs Explained

Cosmos DB Foundry Agent Service: Bring-Your-Own Thread Storage

Posted on September 16, 2026 by steefjan1970

Post 5 of 6 on Cosmos DB Foundry Agent Service integration, owning the thread store instead of leaving it opaque behind a managed API.

Posts 1 through 4 assumed you manage the Cosmos DB account directly. Foundry Agent Service changes that assumption by default: spin up an agent the basic way, and Microsoft manages the thread store for you, out of reach of a direct query. Standard agent setup flips that around. Cosmos DB Foundry Agent Service integration lets threads, system messages, and agent metadata land in a Cosmos DB account you own, sitting right where the schema, search, and checkpointing patterns from the rest of this series already apply.

Why Bring-Your-Own Thread Storage Matters

Data residency, security review, and auditability all get harder when a vendor holds conversation history in a store you can’t query. Standard setup solves that by provisioning three customer-owned resources instead of one managed black box: Azure Storage for uploaded files, Azure AI Search for the agent’s vector stores, and Azure Cosmos DB for everything: conversational messages, threads, and agent metadata. Cosmos DB carries the load that matters most for this series: it’s where the actual conversation lives.

Inside enterprise_memory: Cosmos DB Foundry Agent Service Containers

Standard setup names the resulting database enterprise_memory, and container names inside it depend entirely on which Foundry Agent Service runtime the agent runs on. Foundry Agent Service (Classic) writes to three containers: thread-message-store for end-user conversation messages, system-thread-message-store for internal system messages, and agent-entity-store for agent metadata like instructions and tools. Foundry Agent Service (New) writes to two different containers instead of agent-definitions-v1 and run-state-v1, and neither runtime reads the other’s containers. Check thread-message-store for an agent running on the New runtime, and it comes back empty, not because BYO thread storage failed, but because the data landed somewhere else entirely.

Provisioning Standard Agent Resources

Provisioning means more than a Cosmos DB account on its own. Standard setup also expects an Azure Storage account, an Azure AI Search resource, and an Azure Key Vault for secrets, alongside a deployed agent-compatible model. Once those exist, Microsoft’s Bicep template accepts the resource IDs of existing accounts and wires up the rest: account and project connections, role assignments, and the capability hosts that tell Foundry where agent state actually lives

Two things catch people off guard here, so it’s worth flagging both before you deploy. First, throughput: your Cosmos DB account needs at least 3,000 RU/s total 1,000 RU/s for each of the three baseline containers and that number scales up with every additional project sharing the account, since each project gets its own container set. Undershoot it, and the deployment doesn’t fail quietly; it throws CapabilityHostProvisioningFailed during the capability host step. Second, roles: the project’s managed identity needs Cosmos DB Operator at the account level to provision containers, plus Cosmos DB Built-in Data Contributor at the database level for enterprise_memory. The database-level scope covers every container inside it, so a single role assignment handles the whole set instead of one per container.

Querying Thread History Directly

This is what bring-your-own thread storage actually buys over the default: a direct line into conversation history that Foundry’s own API doesn’t expose. Once a thread exists in thread-message-store (Classic) or run-state-v1 (New), you can run the same vector, full-text, and hybrid queries from post 3 against it — same RANK RRF(...) syntax, same partition-scoped WHERE clause. The only difference is the container: Foundry manages it instead of your own code.

Treat this as a connection into Foundry Agent Service, though, not a replacement for it. Foundry still owns thread creation, run orchestration, and tool invocation; Cosmos DB Foundry Agent Service integration only changes where the resulting data sits and who can query it directly.

Pitfalls

Checking the wrong container set. The single most common source of “BYO thread storage isn’t working” reports is querying Classic’s containers for an agent running on the New runtime, or vice versa. Confirm which runtime a project uses before assuming a missing thread means a broken connection.

Underprovisioning throughput for multiple projects. The 3,000 RU/s floor covers one project’s container set. Add a second project to the same Cosmos DB account, and the container count and the RU/s requirement under it doubles. CapabilityHostProvisioningFailed almost always traces back to this, not to a misconfigured connection.

Assuming you can edit a capability host after creation. You can’t. Pointing a project capability host at the wrong Cosmos DB resource ID means deleting and recreating the project, not patching the connection — worth getting right on the first deployment rather than treating it as a setting to adjust later.

Next: Cost, Caching, and Production Pitfalls

That settles Cosmos DB Foundry Agent Service integration for readers who need full ownership of thread storage rather than a managed default. The last post in this series pulls back to the practitioner-notes view: RU cost drivers, semantic caching, and a closing pitfalls roundup across everything posts 2 through 5 have covered.

Sources

Microsoft Learn — Integration with Foundry Agent Service
Microsoft Learn — Set up standard agent resources for Foundry Agent Service.
GitHub — Standard agent setup Bicep template
Azure Cosmos DB Blog — Azure AI Foundry Connection for Azure Cosmos DB and BYO Thread Storage

Five RAG Architectures in Real Azure Code

Posted on September 13, 2026 by steefjan1970

Over the past few months I kept running into the similar looking infographics, in one form or another: five or six boxes, each a named RAG architecture, arrows showing how a query flows through it. Hybrid RAG. GraphRAG. Agentic RAG. Corrective RAG. Multimodal RAG. They’re useful as vocabulary. They are not implementation guides. None of them show you the part that actually takes the time.

So I built all five RAG architectures, on Azure, against one shared corpus and one shared set of test questions, and measured what came out. This post is the result: what each diagram leaves out, what the equivalent Azure code actually looks like, where the real deployment pitfalls were, and a comparison table built from real runs, not from argument.

The corpus is a fictional Dutch health insurer, Zorgverzekeraar Meridiaan, the same one I’ve used in a couple of other posts in this series. Nine documents: dental and physiotherapy policies, a provider network, an authorization process, a member complaint and the quarterly report that restates it, a stale FAQ sitting next to the current policy, a reimbursement table, and a scanned claim form. Fifteen questions, tagged by which pattern they were designed to stress. All five patterns answer all fifteen questions, so the comparison is apples to apples.

The repo is at github.com/steefjan1/five-rag-patterns if you want to run it yourself.

What the diagram shows vs. what the Azure code does

Hybrid RAG

The diagram draws dense and sparse retrieval as two separate paths that merge into a box labeled Reciprocal Rank Fusion. That box is mostly a non-event on Azure. Azure AI Search’s hybrid query type takes a vector query and a text query together and fuses them server-side. There is no RRF code to write.

What actually takes engineering effort is the index schema: chunk granularity (I chunk by document section, not by a fixed token window, so a retrieval unit is a coherent answer, not an arbitrary slice), which fields are filterable versus searchable versus vector, and whether semantic ranking is worth its cost on top of the fusion you already get for free.

Measured: recall 1.00 across all fifteen questions, the best of any pattern on pure coverage. Precision sits at 0.38, diluted by a fixed top-5 retrieval regardless of how many documents a question actually needs. Cheapest sane baseline in the set: $0.0026 and 2.00 seconds per query.

GraphRAG

The diagram draws one static graph: entities, edges, a subgraph retrieval step, a box for community summaries. What it doesn’t draw is that the graph has a maintenance cost. Community detection (Louvain, via networkx, which runs fine at this corpus’s scale without a dedicated graph database) and community summarization are real compute and real Azure OpenAI spend, paid once at setup and again every time the graph changes enough to shift community boundaries. Nothing about that shows up in the box-and-arrow version.

Entity linking here is a cheap substring match against entity names, not an embedding call, which is part of why this pattern is the cheapest per query in the whole set. Retrieval is a graph walk: two hops, both directions, so a question like “which hospital did this referral come from, and which GP group refers into that hospital” resolves correctly even though no single document states the answer. It’s two separate edges, walked in sequence.

Measured: recall 1.00 on its own three relational questions, the two-hop case included, and 0.21 on the other twelve. No other pattern swings that hard between its own territory and everything else. $0.0013 per query, the cheapest pattern here, in the narrowest lane.

Agentic RAG

The diagram shows a planner routing to tools and a reasoner that loops “until confident.” There is no upper bound drawn anywhere on that loop. Left alone, that is a cost leak, not a reliability feature, so the actual implementation caps it at five iterations and reports hitting the cap as its own outcome rather than quietly forcing an answer and calling it clean.

The other thing worth knowing if you’re building this on Azure: the AI Foundry Agent Service SDK bypasses API Management for its own LLM calls. I found this the hard way on an earlier project in this series. If your governance model depends on APIM, that means routing tool-calling agents through the standard OpenAI SDK pointed at the gateway, not through the framework’s own agent runtime, or every rate limit and kill switch you built stops applying the moment the agent framework makes the call instead of your code.

Two of this pattern’s four tools aren’t retrieval at all. The dental waiting-period and annual-maximum arithmetic is transcribed from the policy documents as plain code, not left for a language model to compute from prose. Insurance eligibility math is exactly the kind of thing an LLM gets subtly wrong under pressure, and exactly the kind of thing code gets right every time.

Measured: precision 1.00, recall 1.00 on its own two questions, and it’s the only pattern that doesn’t collapse elsewhere: recall 0.85 on the other thirteen, because it always has a general search tool as a fallback when nothing more specific fits. That’s the real finding here. It’s not that Agentic RAG is “better,” it’s that it hedges.

Corrective RAG

The diagram shows retrieve, grade, then three branches: answer, rewrite the query and loop back, or fall back to a web search. The rewrite loop has an arrow pointing backward and no stated exit condition. A closed corpus also has no web to fall back to, so “incorrect” here means declining to answer rather than guessing.

The corpus has a document built specifically to test the grading step: an archived FAQ with a plausible, wrong number sitting right next to the current policy with the right one. A pattern with no grading step retrieves both and may cite either. This one grades the retrieval, asks the model to identify which passage is authoritative using published dates and explicit supersession language, and only feeds the authoritative passages to the final answer. What got fetched and what got used are tracked separately on purpose, so a working grader shows zero distractor citations even though the distractor was retrieved.

Measured: precision 1.00, recall 1.00 on the three distractor questions, confirmed live, not just in the design. The more interesting number is that its other twelve questions score better (precision 0.79) than its own target slice (0.67). The grading discipline isn’t just catching the one distractor it was built to catch, it generalizes. That comes at a real cost: 3.97 seconds average latency, roughly double every other pattern, and the highest cost per query in the set, because a full run can mean three model calls instead of one.

Multimodal RAG

The diagram’s box says “shared multimodal embedding model (e.g. CLIP or ColPali),” which means self-hosting an embedding model. That’s a heavier operational commitment than anything else in this comparison needs, and it’s avoidable. This uses caption-then-embed instead: Document Intelligence extracts the actual structure of the reimbursement table (tables are exactly where a vision model hallucinates a plausible-looking row that isn’t in the source, so that step doesn’t get skipped), the vision-capable chat deployment captions the scanned claim form directly, and both captions get embedded with the same text-embedding-3-large deployment every other pattern uses. Same index Hybrid RAG built, two more documents in it, no new index and no new field.

One implementation note that cost real iteration: a first version of the captioning prompt asked for verbatim transcription, which correctly produced the form’s Dutch date format and Dutch status text. A validation step checking the caption against a hand-written ground truth flagged that as a mismatch, because the ground truth expected ISO dates and English. That’s not a captioning bug, it’s a prompt that needed to ask for normalization, not transcription. Worth deciding on purpose, since a shared index with mixed date formats and mixed languages retrieves worse than a normalized one.

Measured: recall 1.00 across the board and groundedness 1.00 on its own two questions, with no degradation on the other thirteen. That composability is the finding: this pattern is Hybrid RAG’s exact retrieve-and-answer loop plus two documents, and the numbers confirm that composition was free.

The comparison table

Same corpus, same fifteen questions, one pass, all five RAG architectures.

Pattern	Avg latency	Cost/query	Overall precision	Overall recall	Own-target recall
Hybrid	2.00s	$0.0026	0.38	1.00	1.00 (n=5)
GraphRAG	2.32s	$0.0013	0.20	0.37	1.00 (n=3)
Agentic	2.09s	$0.0045	0.45	0.87	1.00 (n=2)
Corrective	3.97s	$0.0051	0.77	0.97	1.00 (n=3)
Multimodal	2.18s	$0.0027	0.38	1.00	1.00 (n=2)

“Own-target” means the small subset of the fifteen questions each pattern was actually designed to answer (GraphRAG’s two-hop provider questions, Corrective RAG’s stale-document case, and so on). Every pattern hits recall 1.00 in its own lane. What separates them is what happens outside it: GraphRAG falls to 0.21 recall on the other twelve questions, Agentic RAG only falls to 0.85, and Hybrid, Corrective, and Multimodal don’t fall at all, because their retrieval isn’t scoped to a narrow entity set in the first place.

Two honest caveats on this table. Precision across every pattern is capped low by a fixed top-5 retrieval regardless of how many documents a question actually needs, so precision here measures retrieval breadth more than answer quality, read recall and the own-target column as the more meaningful columns. And the cost figures come from a placeholder price table, not a live Azure billing export, useful for comparing patterns against each other, not for a procurement conversation.

Deployment pitfalls

Every one of these was a real failure against a live Azure subscription, not a hypothetical.

Pinned model versions rot. A deployment written against gpt-4o-mini version 2024-07-18 failed eight months later with ServiceModelDeprecated. The fix wasn’t a newer pin, it was to stop pinning: leave the deployment’s model version empty and let Azure resolve the current default, and check az cognitiveservices model list -l <region> -o table before assuming a model name is still offered at all.

The account kind changed. Azure OpenAI is now provisioned through Foundry as kind: 'AIServices', not the older kind: 'OpenAI'. Same deployment mechanism underneath, different account kind and a newer API version. A template written against the old kind fails Cognitive Services preflight validation, not at compile time.

A malformed policy XML fails at ARM validation, not at Bicep build time. An APIM policy embedded as a Bicep string had a raw double-quoted path literal sitting inside an already double-quoted XML attribute. bicep build compiled it clean, because Bicep has no way to know a string is meant to be well-formed XML. The actual break only showed up against the live ARM validation API, after Azure AI Search, Cosmos DB, and the Foundry account had already finished provisioning. A small script that compiles the template and separately parses every embedded policy string as XML catches this before the next azd up, not during one.

RBAC role assignments alone don’t turn on Azure AD authentication. Azure AI Search kept returning a flat 403 on every data-plane call despite two correctly scoped role assignments, because the service still only accepted API-key authentication. Nothing had told it to accept AAD tokens at all. The fix is a separate property, disableLocalAuth: true, on the search service itself. If a resource with roles that look correct still refuses an authenticated caller, check the resource’s own auth settings before re-checking the role assignment.

Where none of this is the answer

None of these five patterns is the right first move for a small, stable knowledge base. Plain vector search, no fusion, no graph, no grading, no agent loop, is the correct answer until you can name the specific failure mode you’re buying insurance against. Every pattern here is a bet against one kind of failure, and every bet has a cost attached whether or not you ever collect on it.

Don’t build all five for one real system either. Pick based on the failure mode your domain actually has. GraphRAG only pays for itself if your questions are genuinely relational, multi-hop, the kind no single document answers. If they’re not, you’re paying setup cost and getting a narrower Hybrid RAG. Agentic RAG’s flexibility costs a planning call before any retrieval happens at all, worth it if your questions genuinely vary in shape, wasted overhead if they don’t. Corrective RAG’s discipline costs roughly double the latency of everything else in this comparison. That’s a fine trade when a wrong answer is expensive and a two-second wait isn’t. It’s a bad trade for a chat widget where speed is the product.

What this actually proves

The infographic’s taxonomy is real. These five RAG architectures are genuinely different, with genuinely different failure modes, and that part of the diagram holds up. What doesn’t hold up is the implication that the hard part is choosing between them. The hard part, in every case, was the piece the diagram didn’t draw: RRF turned out to be free because Azure AI Search already does it, but community detection is not free and has to be redone as the graph changes. An agent loop needs a hard cap or it’s an open-ended bill. A query rewrite loop needs the same cap for the same reason. A self-hosted multimodal embedding model turned out to be avoidable entirely, caption-then-embed onto infrastructure you already have gets you most of the way there.

The single most useful number in this whole exercise might be the smallest one: Corrective RAG’s grading step scored better on questions it wasn’t built for than on the one it was. That’s a pattern worth paying attention to. The things that make a RAG system more disciplined in one specific place often make it more disciplined everywhere, not just in the place you were testing for.

If you want to see the actual failure modes up close rather than the aggregate table, the earlier posts in this series go deeper on two of them: what naive RAG diagrams leave out covers the hybrid retrieval and groundedness gaps in more detail, and choosing between RAG, GraphRAG, and Agentic RAG when auditability is the constraint makes the conceptual case this post backs with numbers.

The full repo, including the corpus, the eval harness, and every pattern’s implementation, is at github.com/steefjan1/five-rag-patterns.

Multi-Agent State and Checkpointing with Cosmos DB

Posted on September 10, 2026 by steefjan1970

Post 4 of 6 on Cosmos DB multi-agent state coordinating what several agents know about the same conversation, without a separate message bus.

Post 3 settled retrieval for a single agent working alone. This post is about what changes once a second agent enters the picture. Coordinating what several agents know about the same conversation turns out to be a different problem from storing and retrieving one agent’s memory, and Cosmos DB multi-agent state ends up resting on two mechanisms this series already covered: hierarchical partitioning from post 2, and change feed from post 1’s retail monitoring callback.

Shared but Separable: What Changes with Multiple Agents

A single agent needs one memory scope. Moreover, a multi-agent system needs two at once: shared memory that every agent can read and write for coordination, and private memory that lets each agent keep its own persona, prompts, and reasoning history separate from the others. Lose the separation, and agents start bleeding into each other’s context. Lose the sharing, and they can’t coordinate at all.

A triage agent, a product agent, and a specialist agent a common pattern in production multi-agent apps each hold their own scoped state. Still, all three write to the same underlying container, so any of them can pick up where another left off.

LangGraph Checkpointing on Cosmos DB Multi-Agent State

LangGraph’s checkpoint interface persists a graph’s state after every step, and Cosmos DB has more than one implementation of it: langgraph-checkpoint-cosmosdb on PyPI, and the checkpoint saver that ships inside langchain-azure-cosmosdb. Both plug into the same standard LangGraph pattern: compile the graph with a checkpointer, then pass a thread_id on every invocation:

			
from langgraph. graph import StateGraph
from langgraph_checkpoint_cosmosdb import CosmosDBSaver
checkpointer = CosmosDBSaver(
    endpoint=cosmos_endpoint,
    key=cosmos_key,
    database_name="agentmemory",
    container_name="checkpoints",
)
graph = StateGraph(AgentState)
# add_node / add_edge calls wire up triage -> specialist routing here
app = graph.compile(checkpointer=checkpointer)
config = {"configurable": {"thread_id": "contoso:thread-1234"}}
app.invoke({"messages": [...]}, config=config)

		

Encode tenantId:threadId into the thread_id string, and the checkpointer’s hierarchical partitioning lines up with the [tenantId, threadId] partition key from post 2 — the same pattern manages per-user, per-session state at scale, this time for graph checkpoints instead of turn-based memory items. Microsoft’s own multi-agent-langgraph sample builds a personal-shopper scenario on exactly this foundation: a triage agent routes requests, and a product agent answers them using retrieval-augmented generation against the same Cosmos DB account.

Change Feed as the Handoff Mechanism

Post 1 covered change feed as the primitive behind a 2023 retail monitoring solution, a new record in Cosmos DB firing a Function that could raise an incident. The same primitive coordinates agent handoffs: one agent writes a turn, a Function listening on the container’s change feed picks it up, and it hands the conversation to whichever agent should act next. No polling loop checks for new work; the write itself is the signal.

A minimal handoff trigger, using the turn-based schema from post 2:

python

			
import azure.functions as func
def main(documents: func.DocumentList) -> None:
    for doc in documents:
        if doc.get("targetAgent") == "specialist":
            notify_specialist_agent(doc["threadId"], doc["turnIndex"])

		

The triage agent sets targetAgent on the turn it writes; the Function reacts to that write and wakes the specialist agent for that thread.

A Second Worked Example: Spring AI for Java Shops

Python and LangGraph aren’t the only path here. Spring AI 2.0 shipped with a Cosmos DB-backed vector store and memory integration for Java, and Microsoft’s multi-agent-spring-ai sample mirrors the LangGraph pattern in Java: multiple agents, one Cosmos DB account, the same shared-but-separable memory shape. Worth a look if the rest of the stack runs on the JVM rather than Python.

Pitfalls

Shared containers without tenant or session isolation. A checkpoint container that mixes every tenant’s graph state leaks context across customers the moment a partition or vector index goes unsharded; the same isolation failure post 3 flagged for vector search is now showing up in agent state instead of retrieved memories. Apply the same [tenantId, threadId] discipline to checkpoints that post 2 applied to turns.

Treating change feed as globally ordered. Change feed guarantees order within a single partition key, not across the whole container. Moreover, a handoff design that assumes “the Function always sees writes in the exact order they happened across every agent” breaks the moment two agents write to different partitions at close to the same time. Design handoffs so each step only depends on ordering within its own thread’s partition, not on a global sequence that Cosmos DB never promised.

Next: Cosmos DB Inside Microsoft Foundry Agent Service

That covers Cosmos DB multi-agent state when you manage the account directly. Post 5 covers the other path: Microsoft Foundry Agent Service’s bring-your-own thread storage, where Cosmos DB still does the work, but Foundry owns the orchestration layer on top of it.

Sources

GitHub — AzureCosmosDB/multi-agent-langgraph
GitHub — AzureCosmosDB/multi-agent-spring-ai
PyPI — langgraph-checkpoint-cosmosdb
Microsoft Learn — AI agents in Azure Cosmos DB
Your own archive — Leveraging Azure Cosmos DB for End-to-End Monitoring of Retail Processes
GitHub — steefjan1/cosmos-agent-memory-lab — the runnable sample behind this checkpointer and handoff, and behind posts 2 and 3

Finding the Right Memory: Vector, Full-Text, and Hybrid Search in Cosmos DB

Posted on September 3, 2026 by steefjan1970

Post 3 of 6 on Cosmos DB agent memory search, because storing memory well doesn’t guarantee you’ll retrieve the right piece.

Post 2 ended with the schema settled and retrieval still open. This post closes that gap: the practical mechanics of Cosmos DB agent memory search, one container, four query patterns. Run the same question four different ways against that container, and it comes back with four different answers, because “find the right memory” isn’t one query pattern; it’s at least three, and knowing which one to reach for is most of the job.

Vector Indexing for Cosmos DB Agent Memory Search

Cosmos DB supports two vector index types, and the right one depends almost entirely on how many vectors you’re searching, not on anything specific to agents.

quantizedFlat compresses each vector and scans the compressed space exactly. It suits smaller workloads (tens of thousands of vectors) and trades a small amount of accuracy for lower RU cost and faster scans. For a single tenant’s short-term memory, this is often enough on its own.

DiskANN, on the other hand, indexes vectors for approximate nearest-neighbor search and scales to hundreds of thousands or billions of embeddings, with dynamic updates and strong recall even at that size. Post 1 already leaned on DiskANN as part of the case for Cosmos DB as a unified store; this is the mechanism behind that claim.

Sharding the Vector Index for Multitenant Isolation

DiskANN doesn’t have to search across every vector in the container. A vectorIndexShardKey partitions the index itself by a property you choose: session, user, or tenant, so a query only searches candidates within that shard instead of the whole container.

That maps directly onto the partition key work from post 2: set the vectorIndexShardKey to tenantId, or to the same [tenantId, threadId] pair you already use as the partition key, and semantic search for one tenant never touches another tenant’s vectors. A global, unsharded index still works and makes searching everything at once simpler, but it’sonly appropriate for a single-tenant app or a genuinely shared knowledge base where cross-tenant recall is the point rather than a leak.

Full-Text Search: When Precision Beats Semantics

Vector search finds what’s semantically similar. Sometimes semantically similar isn’t what you want — a customer asking about “the refund policy” needs the actual refund policy language, not five conceptually related passages about returns in general.

Full-text search on Cosmos DB handles that case through BM25, a statistical ranking function that scores by term frequency and document length. Cosmos DB applies linguistic processing automatically: tokenization, stemming, case normalization, so “running” still matches “run” or “ran.” It’s the right tool whenever exact terms or phrases carry meaning that a vector embedding would blur.

Hybrid Search: Combining Both with RRF

Most agent memory queries don’t need to choose between semantic and lexical relevance; they need a blend of both. That’s what Reciprocal Rank Fusion (RRF) does: it takes the vector-similarity ranking and the BM25 ranking for the same result set and merges them into one combined rank, instead of forcing a pick between the two.

In practice, this shows up as a single ORDER BY RANK RRF(...) clause, which the next section demonstrates directly.

Four Ways to Ask the Same Question

Take the turn-based schema from post 2 — tenantId, threadId, turnIndex, messages, embedding, content and run the same underlying question against it four ways. (content is a flat, denormalized copy of the turn’s text, added specifically because Cosmos DB doesn’t support wildcard array paths like /messages/*/content in a full-text policy or index the full-text and hybrid queries below point at c.content rather than c.messages for exactly that reason.)

Most recent, by recency:

			
SELECT TOP 5 c.messages, c.turnIndex
FROM c
WHERE c.tenantId = @tenantId AND c.threadId = @threadId
ORDER BY c.turnIndex DESC

Semantic, by vector similarity:

			
SELECT TOP 5 c.messages, VectorDistance(c.embedding, @queryVector) AS score
FROM c
WHERE c.tenantId = @tenantId AND c.threadId = @threadId
ORDER BY VectorDistance(c.embedding, @queryVector)

Hybrid, blending both with RRF:

			
SELECT TOP 5 c.messages, VectorDistance(c.embedding, @queryVector) AS score
FROM c
WHERE c.tenantId = @tenantId AND c.threadId = @threadId
ORDER BY VectorDistance(c.embedding, @queryVector)

Keyword, by exact phrase:

			
SELECT TOP 5 c.messages, c.turnIndex
FROM c
WHERE c.tenantId = @tenantId AND c.threadId = @threadId
  AND FULLTEXTCONTAINS(c.content, @phrase)
ORDER BY c.turnIndex DESC

		

Run all four against a thread where a customer asked about refunds three times, in different words, across twenty turns, and the differences stop being theoretical fast: recency surfaces whichever turn happened most recently, even if it’s off-topic; semantic search pulls in every conceptually related turn, including the ones that used different words entirely; hybrid balances the two; keyword search returns only the turns that used the customer’s actual phrase, and ranks them by recency underneath that filter.

Running These Queries in Data Explorer

The four queries above use parameterized SQL, the same form search.py, from the companion repo behind this series, sends through the Python SDK, which binds @tenantId, @queryVector, and @phrase properly before the query runs. Paste them as-is into the Azure Portal’s Data Explorer query pane instead, and two things break, neither of which is a schema or code bug:

Data Explorer’s query box doesn’t bind named parameters. A query that leaves @tenantId unresolved either matches nothing and returns “No results” silently, or for VectorDistance() inside ORDER BY and FullTextScore() fails to compile outright, because both functions require their arguments to resolve to literal values at query-compile time rather than at execution time.

Swap every @parameter for a literal value and all four run cleanly. Against the seeded sample data (tenantId = "contoso", threadId = "thread-1234", searching for "refund"):

Recency, with literals:

			
SELECT TOP 5 c.messages, c.turnIndex 
  FROM c WHERE c.tenantId = "contoso" AND c.threadId = "thread-1234" 
  ORDER BY c.turnIndex DESC

Semantic, with literals:

			
SELECT TOP 5 c.messages, VectorDistance(c.embedding, [0.8196, 0.6392, -0.2471, 0.1608, -0.8667, 0.4902, -0.2549, 0.2235]) AS score
FROM c
WHERE c.tenantId = "contoso" AND c.threadId = "thread-1234"
ORDER BY VectorDistance(c.embedding, [0.8196, 0.6392, -0.2471, 0.1608, -0.8667, 0.4902, -0.2549, 0.2235])

Hybrid, with literals:

			
SELECT TOP 5 c.messages, c.turnIndex
FROM c
WHERE c.tenantId = "contoso" AND c.threadId = "thread-1234"
ORDER BY RANK RRF(
    VectorDistance(c.embedding, [0.8196, 0.6392, -0.2471, 0.1608, -0.8667, 0.4902, -0.2549, 0.2235]),
    FullTextScore(c.content, "refund")
)

		

Keyword, with literals:

			
SELECT TOP 5 c.messages, c.turnIndex
FROM c
WHERE c.tenantId = "contoso" AND c.threadId = "thread-1234"
  AND FULLTEXTCONTAINS(c.content, "refund")
ORDER BY c.turnIndex DESC

		

Pitfalls

Reaching for DiskANN on a small dataset. DiskANN’s approximate search and sharding options solve a scale problem. Below roughly ten thousand vectors, quantizedFlat gets equivalent recall for less operational complexity and lower RU cost. Default to DiskANN because it sounds like the “serious” choice, and you’ve added index-shard decisions to a workload that never needed them.

A global vector index in a multitenant app. Skip the vectorIndexShardKey, and a semantic query searches every candidate in the entire container, tenant boundaries or not. Nothing stops the query from surfacing another tenant’s conceptually similar memory in the result set unless a WHERE clause happens to filter it back out after the fact, and relying on a filter to catch what the index itself should have scoped is the kind of gap that shows up in an audit, not in testing.

Forgetting WHERE filters still apply. Vector and hybrid queries look like they replace normal filtering, but ORDER BY VectorDistance(...) or ORDER BY RANK RRF(...) still runs inside a WHERE-scoped query, same as any other. Leave the WHERE c.tenantId = @tenantId AND c.threadId = @threadId clause off a semantic query, and it searches everything the container holds, not just the thread the agent is currently in.

Next: Coordinating Multiple Agents

That settles Cosmos DB agent memory search for a single agent working alone. Coordinating what several agents know about the same conversation is a different problem, and it’s where change feed, a mechanism post 1 already covered as a callback to the 2023 retail monitoring work, comes back to tie multi-agent state together. That’s post 4.

Sources

Microsoft Learn — Agent memories in Azure Cosmos DB for NoSQL
Microsoft Learn — Vector search in Azure Cosmos DB
Microsoft Learn — Sharded DiskANN (vectorIndexShardKey)
Microsoft Learn — Hybrid search in Azure Cosmos DB
GitHub — steefjan1/cosmos-agent-memory-lab — the runnable sample behind these four queries, and behind posts 2 and 4

The Microsoft AI Stack in 2026 and the Certification Trail That Runs Through It

Posted on September 2, 2026 by steefjan1970

I spent some time observing what’s inside the Microsoft AI stack. Foundry, Agent Framework, Logic Apps, AI Search, Purview, Entra. After a while, I had a picture of how the pieces fit and decided to draw one myself to share.

Then recently Microsoft published AI-500, an expert certification for multi-agent systems. That made me curious whether Microsoft’s view of the platform matches my own. So I plotted the certification trail against my diagram. This post shows the result.

The Microsoft AI stack as I see it

Six layers. Five of them stack vertically. The sixth runs down the side, through all the others.

Models sit at the bottom: GPT-5, Claude, Mistral, Grok, Microsoft’s own MAI and Phi, Llama, DeepSeek, and the open catalog. This is the least differentiated layer. In Foundry, swapping one model for another is a configuration change. It gets the most attention and deserves the least.

Infrastructure comes next. Foundry is the hub, alongside Azure OpenAI, Azure ML, AKS, Container Apps, App Service, and Foundry Local for the edge. This is hosting, serving, and compute. Solid and well understood.

Data and context: the real moat

I split the data layer in two, because it hides the most important part of the platform. Layer 3a holds the sources of truth: Microsoft Graph, SharePoint, Exchange, Fabric and OneLake, Dataverse, and the vector stores in Cosmos DB, Azure SQL, and PostgreSQL.

Layer 3b is context. Work IQ, Fabric IQ, Foundry IQ, and Azure AI Search turn enterprise data into grounding that respects who is asking. Permission-trimmed retrieval must honor Entra ACLs at query time, not filter after ranking. AI Search supports this through document-level access control. Get it wrong and answers leak, or recall collapses.

This is where the hard engineering hours go. Swapping a model is a config change. Graph-grounded context is months of work.

Agents, Copilots, and the layer vendors leave out

The agentic platform sits above the data. Microsoft Agent Framework merged Semantic Kernel and AutoGen. Next to it sit Foundry Agent Service, Copilot Studio, Logic Apps, Azure Functions, Service Bus, and Event Grid. MCP and A2A handle tools and agent-to-agent communication.

The Copilot layer is on top: Microsoft 365, GitHub, Security, Dynamics 365, Power Platform, and Teams. This is distribution. These are the surfaces people already live in. As a result, enterprise AI adoption is easier when data, permissions, infrastructure, and applications already exist in one ecosystem.

Then comes the sixth layer of the Microsoft AI stack, the one vendor diagrams leave out: governance and evidence. Entra ID and Entra Agent ID handle identity. Purview covers labels, DLP, and audit. Content Safety and API Management provide guardrails and the AI gateway. Defender, Sentinel, Azure Monitor, Log Analytics, and Azure Policy deliver traces, evaluations, and control evidence.

I work for a regulated organization. Before anything goes live, Legal and Internal Audit ask four questions. Who approved the agent? What data did it access? Which controls applied? What happened when it made a wrong decision? Distribution makes deployment easier. However, without traceable evidence it does not make the system production ready. That is why this layer runs vertically through my drawing.

The Microsoft AI certification trail

I knew AI-900. I had not followed what replaced it. This week I learned that Microsoft now offers a full set of AI certifications, from fundamentals to an expert exam. The expert exam, AI-500, is in beta. I read the study guide. Its scope says a lot: orchestration patterns, agent-to-agent protocols, observability, guardrails, and cost control. Architecture work, end to end.

What made it click was laying the whole trail side by side. Each step has its own verb.

AI-901, Azure AI Fundamentals. Understand the concepts and services.
AI-103, AI Apps and Agents Developer. Build applications and agentic solutions on Foundry.
AI-200, Azure AI Cloud Developer. Engineer cloud-native AI properly.
AI-300, ML Operations Engineer. Operate models in production.
GH-300 and GH-600, Copilot and Agentic AI Developer. Ship with agents working beside you.
AI-500, Multi-Agent AI Solutions Expert. Orchestrate systems of agents at scale.

Plotting the trail on the Microsoft AI stack

Six exams, six layers of the Microsoft AI stack. I mapped each exam to its layers, in its study guide exercises, and its center of gravity.

The fundamentals exam touches every layer at concept depth. AI-103 lives in models, infrastructure, context, and the agentic platform. AI-200 moves down into infrastructure and data. AI-300 sits on infrastructure and the evidence plane, because operating models in production is mostly monitoring and evaluation. Meanwhile, the GitHub exams live at the top, where developers meet agents in the editor.

AI-500 is the interesting one. It spans context, the agentic platform, and governance. It does not test models at all. In fact, three of its five headline topics belong to the plane most stack diagrams leave out.

That was the moment the two pictures agreed. My diagram says the hard part of the Microsoft AI stack is context and evidence, not model choice. Microsoft’s expert exam tests context and evidence, not model choice. I did not expect a certification roadmap to confirm an architecture opinion, but here we are.

Where this is the wrong answer

Do not read the trail as a ladder you must climb in order. If you already run agents in production, AI-500 reflects your work, and AI-901 will teach you nothing. Platform engineers should look at AI-200 and AI-300 first. Developers should start with the GitHub exams.

Also, do not read my layer mapping as Microsoft’s. It is my reading of the study guides, and beta study guides move.

Finally, do not confuse the certificate with the evidence. Passing AI-500 shows you know what a control looks like. It does not produce the audit trail for your agent. That still takes Purview configured, Entra Agent ID issued, traces flowing to Log Analytics, and someone signing off. The exam is a map of the work. The work is still the work!

Credits

The Azure icons come from the official Azure architecture icon set. Product marks belong to their owners. Microsoft’s announcements are on the Skills Hub blog: Multi-Agent AI Solutions Expert, AI Apps and Agents Developer Associate, and GitHub Agentic AI Developer. The stack diagram builds on a five-layer picture that circulated on LinkedIn; the split data layer and the governance plane are my additions.

Designing a Cosmos DB Agent Memory Schema

Posted on September 1, 2026 by steefjan1970

Post 2 of 6 on Cosmos DB agent memory schema design before you write a line of agent code.

Post 1 made the case for one database instead of three. This post is about the decisions that determine whether that one database actually holds up: partition key and item shape, both of which you choose before an agent ever writes a turn. Get these wrong, and no amount of DiskANN or 99.999% SLA saves you from a hot partition or a rewrite-the-whole-item cost curve later.

Partition Keys: The Foundation of a Cosmos DB Agent Memory Schema

Because Cosmos DB automatically partitions data, the partition key is the single most consequential choice in the schema. It decides how writes distribute, how queries scope, and eventually how much a bad decision costs to unwind. Three strategies cover most agent memory scenarios, and each trades distribution for locality differently.

Partition Keys

GUID as the partition key. Every item lands in its own logical partition. Writes distribute as evenly as possible, so this works well for high-volume, write-heavy logging where you rarely need to reassemble a conversation later think raw telemetry more than chat history. The cost shows up on read: reconstructing a thread means a cross-partition query.
threadId as the partition key. All turns in a conversation share one partition key, so “give me the last 10 turns” or “vector search within this thread” both stay inside a single partition. This is the default for conversational agents and RAG apps, provided threads are numerous and varied enough to avoid concentrating writes on a few hot values.
[tenantId, threadId] as a hierarchical partition key. This is where the series comes full circle: I covered hierarchical partition keys in Azure Cosmos DB’s Latest Performance Features back in 2023, for the same tenant-then-item pattern. Nothing about the mechanism changed for agent workloads: threads still colocate under their tenant, tenant-level queries still avoid scanning every partition, and Cosmos DB still sub-partitions past the 20 GB logical-partition ceiling the same way it always did. Only the data moving through it is new.

There’s no universally correct answer here, so match the strategy to the query pattern that matters most: threadId if “give me this conversation” dominates, GUID if raw write throughput dominates, hierarchical if tenant isolation is a governance requirement and not just a nice-to-have.

Three Ways to Shape the Memory Item

Partition key decides where data lives. Item shape decides what it costs to read and write once it’s there, and that’s the other half of a Cosmos DB agent memory schema.

One document per turn (recommended default). Each item holds a complete exchange: a user prompt, the agent’s reply, and any tool call in between, and carries threadId, turnIndex, and an embedding alongside it. Most single- and multi-agent apps default to this shape, because it balances a small, cheap-to-write item against enough context to be useful on its own. “Latest N turns” is a simple ORDER BY turnIndex query, and ttl can expire old turns individually instead of touching the whole thread.

One document per response. Every user message, agent reply, and tool result gets its own item, all sharing a threadId. This is the most granular option, useful when you need to embed and search every single utterance independently, but it multiplies item count and RU cost on read. It loses the natural “question and answer together” unit that a semantic cache wants.

One document per thread. The whole conversation lives in one item that grows with every append. Reading the full history is a single read, which sounds appealing until a long-running thread turns every new turn into a full-item rewrite. Treat this as an anti-pattern unless the thread is short and you can bound its length by design: a five-turn onboarding flow, maybe; an open-ended assistant conversation, no.

TTL as Memory Lifecycle Management

Short-term memory should disappear on its own, and time-to-live is how Cosmos DB does that without a cleanup job. Set a default ttl on the container, and every item expires after that many seconds unless it overrides the value itself; set the container default to -1 instead, and Cosmos DB turns on ttl without expiring anything, unless an item sets its own positive ttl field. That second mode is the more useful one for agent memory, since it lets long-term memories (ttl: -1 on the item, meaning never expire) sit in the same container as short-term turns (ttl: 3600, gone in an hour) without a separate container or a background job doing the deleting.

A Worked Schema

Here’s the turn-based model from the previous section as an actual container and item.

Create the container with a hierarchical partition key, and turn on ttl at the container level:

			
az cosmosdb sql container create \
  --account-name my-cosmos-account \
  --database-name agentmemory \
  --name turns \
  --resource-group my-rg \
  --partition-key-path "/tenantId" "/threadId" \
  --ttl -1

		

A single turn, ready to insert:

			
{
  "id": "b9c5b6ce-2d9a-4a2b-9d76-0f5f9b2a9a91",
  "tenantId": "contoso",
  "threadId": "thread-1234",
  "turnIndex": 7,
  "messages": [
    { "role": "user", "content": "What's our refund policy for accessories?" },
    { "role": "agent", "content": "Refund policy is 30 days for unopened items." }
  ],
  "embedding": [0.013, -0.092, 0.551],
  "ttl": 3600
}

		

And writing it with the Python SDK:

			
from azure. cosmos import CosmosClient
client = CosmosClient(url, credential)
container = client.get_database_client("agentmemory").get_container_client("turns")
container.upsert_item({
    "id": "b9c5b6ce-2d9a-4a2b-9d76-0f5f9b2a9a91",
    "tenantId": "contoso",
    "threadId": "thread-1234",
    "turnIndex": 7,
    "messages": [
        {"role": "user", "content": "What's our refund policy for accessories?"},
        {"role": "agent", "content": "Refund policy is 30 days for unopened items."},
    ],
    "embedding": [0.013, -0.092, 0.551],
    "ttl": 3600,
})

		

Swap ttl: 3600 for ttl: -1 on any item you want to keep past the container default as summarized long-term memory; for instance, it survives while the rest of the thread ages out on schedule.

Seeing It Live

The rest of this section shows the same schema running against a real Cosmos DB account, not just described on paper.

The provision.py creating the turns container with the [/tenantId, /threadId] hierarchical partition key and container-level ttl.

The same container in Data Explorer’s Scale & Settings pane partition key and ttl exactly as provision.py set them.

A seeded turn in Data Explorer’s Items view tenantId, threadId, turnIndex, messages, and embedding, matching the JSON above field for field.

Pitfalls

Hot partitions from low-cardinality tenant keys. A hierarchical [tenantId, threadId] key only distributes well if tenants themselves are numerous and reasonably balanced in volume. One enterprise customer generating 80% of total traffic under a single tenantId value creates a hot partition no amount of threadId variety underneath it fixes. Check tenant volume distribution before committing to this key, not after.

Unbounded thread-per-item growth. It’s tempting to reach for one-document-per-thread because “just read the whole conversation” feels simpler in application code. In practice, an item that grows by one append per turn racks up RU cost on every single write as the item gets larger, and Cosmos DB item size limits eventually cap how long a thread can run at all. If a thread’s length isn’t predictable and short, don’t model it this way.

Forgetting embeddings need to be top-level fields. If you nest an embedding array inside a messages object, it won’t be indexable for vector search — it has to sit at the top level of the item, alongside threadId and turnIndex, for the container’s vector policy to pick it up. This one is easy to miss because the item still writes successfully; it just never shows up in a vector query, and that failure is silent until someone notices recall is worse than expected.

Next: Finding the Right Memory

The schema in this post gets data into Cosmos DB efficiently. It doesn’t yet get the right memory back out at the moment an agent needs it — that’s the job of vector, full-text, and hybrid search, which is where post 3 picks up. That settles the Cosmos DB agent memory schema; retrieval is the next problem worth solving properly.

Sources

Microsoft Learn — Agent memories in Azure Cosmos DB for NoSQL
Microsoft Learn — Time to live (TTL) in Azure Cosmos DB.
Microsoft Learn — Hierarchical partition keys
Your own archive — Azure Cosmos DB’s Latest Performance Features
GitHub — steefjan1/cosmos-agent-memory-lab — the runnable sample behind this schema, and behind posts 3 and 4

Don’t Build Around Today’s Model. Build for the AI Control Plane

Posted on August 31, 2026 by steefjan1970

Microsoft isn’t just shipping AI models. It is building toward a model-agnostic AI control plane. The bigger story behind the recent Microsoft Foundry updates is not another model launch. It is the architecture direction. Once you see it, the individual announcements fall into place as layers of one stack:

AI Control Plane → Model Router → Agents → RAG/Knowledge → MCP/Tools → Enterprise Systems

The AI control plane takes shape

Two August updates make the direction concrete. First, the model router update expanded the router to 28 regions for global standard and 21 data zone regions. The supported pool now includes Anthropic Claude Opus 4.8 and the GPT-5.6 family, while deprecated models such as DeepSeek-V3.1 and the gpt-5-chat line were pruned. Second, Foundry added DeepSeek-V4-Flash-0731 and NVIDIA Nemotron 3.5 Lightning to the catalog, each available through multiple deployment paths: Direct from Azure, Fireworks on Foundry, or the Hugging Face collection on managed compute.

I covered the router update in more detail on InfoQ, including the caveats around behavioral stability. Here, I want to focus on what the updates mean for your architecture.

Neither update is spectacular on its own. Together, however, they show a platform where models arrive, improve, and retire underneath a stable endpoint. Your application keeps calling the same integration while the pool refreshes. That is control plane behavior, not model shipping.

Agent ≠ Model

Here is the key idea for enterprise architects: an agent is not a model. An agent is an orchestration unit with instructions, knowledge, and tools. The model is a swappable dependency underneath it.

The model router makes that separation operational. It selects a model per request, optimizing for quality, cost, and latency within the regions your governance allows. You can run it in balanced, quality, or cost mode, and you can restrict routing to an approved subset of models. Moreover, every response includes a model field that shows which model handled the request, so the routing decisions leave an auditable trail.

Microsoft frames this as a hill climb: model selection as a continuous, measured loop rather than a one-time decision. In an ecosystem where the frontier moves monthly, a hardcoded model choice goes stale fast. An agent bound to the control plane instead of a specific model can evolve without a rebuild every time the leaderboard changes.

What this means for your architecture

For enterprise architects, the practical guidance follows directly. Treat model selection as configuration, not code. Put governance, observability, and policy at the control plane layer, because that is where they survive model churn. Ground agents in your own knowledge through RAG, and connect them to enterprise systems through MCP and tools. Consequently, each layer can evolve at its own pace. The stack outlives any single model.

Where this is the wrong answer

Model-agnostic routing is not free, and it is not always right. If your workload requires reproducible behavior, for example in regulated decision flows, a pinned model version beats a router that refreshes its pool automatically. Your evaluations were run against a specific model; a silent pool refresh invalidates them until you re-run. There are operational caveats too: Anthropic models still need to be deployed separately before the router can reach them, and routing modes take time to propagate. Finally, a single well-tuned small model per agent remains the simpler option for narrow, high-volume tasks. Routing adds a layer you must monitor. Only accept that cost when model diversity actually pays for itself.

Closing thoughts

The direction is clear even where the details will shift. Models are becoming interchangeable parts. The durable investment is the control plane: routing, governance, evaluation, and the connective tissue to your enterprise systems. So don’t build around today’s model. Build for the control plane, and let the models come and go.

What’s your approach: one model per agent, or model-agnostic agents?

Why Cosmos DB Ends Up as the Agent Memory Database

Posted on August 30, 2026 by steefjan1970

The first post in a series on Cosmos DB agent memory for AI agents, starting nine years before “AI agent” was a category.

In 2017, I built a proof of concept for a customer: a knowledge base on Cosmos DB, using the Graph model and Search, running at roughly 1,000 euros a month. I presented it at CloudBrew. One attendee wasn’t impressed:

“The most uninteresting talk of the day came from Steef-Jan Wiggers, who, in my opinion, delivered an hour-long marketing pitch for CosmosDB. I think it’s expensive for what it currently offers, and many developers could architect something with just as much performance without needing CosmosDB.”

He wasn’t wrong that 1,000 euros a month raises eyebrows as a line item. He was wrong about what the line item paid for: the knowledge base was the product a subscription business planned to sell. Compare the cost to the revenue it enabled, and it’s negligible. Compare it to nothing, and of course it looks “expensive.” I made the same point about Figma’s AWS bill last year: $109 million a year sounds alarming until you check it against $821 million in revenue and a business model that requires sub-100ms real-time collaboration for 13 million users. In short, cost without context is just a number that sounds big.

So here’s the same argument, nine years later, with a different workload. Agent memory, chat turns, tool call results, embeddings, and user preferences are expensive to store the wrong way and reasonably cheap to store the right way, and increasingly “the right way” means one database instead of three. I’ll come back to the actual RU numbers in post 6; for now, this post is about why the architecture argument holds up before cost even enters the picture.

The Same Shape of Problem, Nine Years Apart

Strip away “AI agent” and look at what you’re actually storing: short-lived, high-volume, time-ordered records that need fast writes and selective recall. That’s chat turns and tool outputs today. It’s also, structurally, what I modeled in a Cosmos DB Conf 2023 talk on end-to-end retail process monitoring, messages and batches flowing between an ERP, a WMS, and a PIM system, which I tracked so a retailer could tell where something broke.

Different domain, same shape, though: append-heavy writes, a need to reconstruct “what happened, in order,” and a downstream system (an incident manager then, an LLM now) that needs the right slice of history on demand, not the whole history every time.

In general, agent memory falls into two categories:

Short-term (episodic/working) memory — the last 5–10 turns of a conversation, intermediate tool call results, partial task state. Useful for the current task, disposable afterward (Cosmos DB’s time-to-live feature is a natural fit here; more on that in post 2).
Long-term memory — user preferences, summarized threads, facts the agent should persist and recall across sessions.

Both need somewhere to live, but the default answer for the last few years has been: somewhere different.

Why the Stitched Stack Breaks Down for Cosmos DB Agent Memory

The common pattern from 2022 through 2025 was to give each concern its own database: an in-memory store for caching and session state, a relational database for operational data and conversation logs, a purpose-built vector database for embeddings. A reasonable instinct, in theory: each tool for its own job.

In practice, though, it doesn’t hold up once an agent is the thing reading and writing across all three, on every turn.

Each piece has a real weakness once agents are the workload, not an afterthought:

Pure vector databases tend to offer no strong read/write guarantees, limited ingestion throughput, availability below 99.9%, a single (eventual) consistency level, and thin multitenancy support. Fine for an embeddings side-project. Shaky as the record of what an agent told a customer.
Relational databases fight the fluid, evolving schema of agent state, new fields, new memory types, and nested tool outputs without migrations and, often, downtime.
In-memory caches are fast and don’t persist, which is exactly the opposite of what long-term memory needs.

As a result, three systems also means three consistency models, three availability profiles, and three places a multi-agent system can silently desynchronize. And that complexity tax doesn’t show up in any single service’s bill, which is part of why it’s easy to miss until something breaks in production.

The Unified Case — and How Much of It I’d Already Used

The pitch for Cosmos DB agent memory as a unified layer rests on a small set of properties: single-digit-millisecond latency, a 99.999% availability SLA on the NoSQL API, DiskANN-based vector indexing built into the same store as the operational data, multi-master writes, and five selectable consistency levels from strong to eventual. In plain terms, that’s one system that’s fast enough for the hot path, available enough for production, and flexible enough to hold embeddings next to the record they came from.

In fact, two of the pieces that make this work aren’t new to me, or new to this blog.

Change feed, for example. In the retail monitoring solution, change feed was the mechanism that turned a write into a trigger: a new record landing in Cosmos DB fired a Function, which could raise an incident. That’s the same primitive I’ll use in post 4 to coordinate handoffs between agents in a multi-agent system: one agent’s write becomes another agent’s signal to act, without polling.

Hierarchical partition keys, likewise. I covered these in Azure Cosmos DB’s Latest Performance Features back in 2023: partitioning by tenant, then by item, to keep related data colocated while avoiding the 20 GB logical partition ceiling. The mechanism hasn’t changed; what’s changed is the workload. Post 2 uses the same [tenantId, threadId] pattern to isolate one customer’s agent conversations from another’s.

Even so, I didn’t build either feature for AI agents. Both turned out to be exactly what agent memory needs a decent sign that the underlying database was solid before the AI use case arrived, and nobody retrofitted it to fit.

Where This Series Is Headed

This post is the framing argument for Cosmos DB agent memory. From here, the rest of the series gets specific:

Post 2 — designing the agent memory schema itself: partition key choice, TTL, and the turn-based data model that works best in practice.
Post 3 — vector, full-text, and hybrid search for recalling the right memories, not just any memories.
Post 4 — multi-agent state and coordination, including change feed as the handoff mechanism.
Post 5 — wiring Cosmos DB into Microsoft Foundry Agent Service as bring-your-own thread storage.
Post 6 — the cost conversation, properly this time: RU drivers, semantic caching, and what this actually costs to run at scale.

The 2017 knowledge base cost 1,000 euros a month and paid for itself many times over as a revenue-generating product. Ultimately, the question worth asking about agent memory infrastructure in 2026 isn’t “is this expensive”; it’s the same question it always was: expensive relative to what?

Sources

Microsoft Learn — AI agents in Azure Cosmos DB
Microsoft Learn — Time to live (TTL) in Azure Cosmos DB.
Your own archive — Azure Cosmos DB’s Latest Performance Features
Your own archive — Figma AWS Costs Explained: Beyond the Hype and Panic
Microsoft Learn Shows — Leveraging Azure Cosmos DB for End-to-End Monitoring of Retail Processes

When Agentic Workloads Break the PaaS Assumptions

Posted on August 27, 2026 by steefjan1970

This series started with a map and grew into seven pieces. Five layers came first: compute, where load shape picks the service; messaging and orchestration, where two questions replace four product choices; data patterns, where idempotency and the outbox keep a platform correct; governance and identity, where policy and audit become the compliance posture; and observability and FinOps, where behaviour and cost become visible. Then came the lens: from design to demonstrable operation, the shift from “is it built?” to “can we operate it responsibly?”

Every one of those pieces rests on a shared assumption. The system does what you told it to do. You wrote the workflow, you defined the routes, you set the policies, and the platform executes them. That assumption has held for every integration platform I’ve built. Agentic workloads break it. So this capstone asks what changes when the thing making decisions inside your platform is a model, not your code, and why each layer, plus the readiness lens itself, deserves a second look because of it.

The assumption agentic workloads break

Conventional integration is deterministic. A message arrives, a workflow runs its defined steps, a router sends it where the rules say. You can read the code and know what will happen. You can test every path. When something fails, you trace it to a step you wrote.

Agentic workloads replace part of that determinism with a model that decides at runtime. The agent reads context, picks a tool, interprets the result, and chooses the next action. Moreover, it does so differently depending on inputs you didn’t fully anticipate. That’s the point of it: the flexibility is the feature. But it means you can no longer read the code and know what will happen. So the ground under every layer shifts: behavior is no longer exactly what you specified.

None of this argues against agentic workloads. It argues for revisiting each layer with the shift named explicitly. Let’s do that.

Compute: the loop changes the shape of the work

The compute layer sorted workloads by load shape: steady request traffic to App Service, event-driven bursts to Functions or Container Apps. Agentic workloads add a shape that sorting didn’t account for: the loop.

An agent doesn’t process a request and return. It reasons, calls a tool, waits, observes, and reasons again, sometimes for many cycles, before it finishes. That’s neither a clean request-response nor a discrete event. Instead, it’s a long-running loop of unpredictable duration with external calls in the middle. So the compute question changes. You’re no longer asking “steady or bursty” alone. You’re asking how to host something that runs for seconds or minutes, holds state across tool calls, and scales on a dimension concurrent reasoning loops that CPU-and-memory autoscale captures poorly. Container Apps with event-driven scaling often fit better here than App Service, and the orchestration frequently belongs in a workflow engine rather than raw compute.

Messaging and orchestration: the agent is a non-deterministic router

The messaging layer drew a clean line. Deterministic routing rules sent messages where the logic dictated. An agent orchestrating tool calls is, in effect, a router too, but a non-deterministic one. It decides which tool to call from its reading of the context, not from a rule you wrote.

The reliability consequences are real. Delivery guarantees still matter; an agent that triggers a business action still needs that action to occur exactly once, so Service Bus and the idempotency store in the data layer remain as relevant as ever. What changes is predictability. You can’t fully anticipate which actions the agent will trigger, or in what order. Therefore, the orchestration has to stay correct under sequences you didn’t design for. One practical lesson from building these loops applies directly: agent outputs rarely arrive as the clean structures a deterministic step would emit, so you build explicit bridges between agent actions rather than assuming shape.

Data: state and correctness under non-determinism

The data patterns held a platform correct when systems it didn’t control misbehaved. Agentic workloads make those patterns more necessary, not less, and they add one more.

Idempotency matters more because an agent may retry a tool call or repeat an action as it reasons, so the dedup store carries a heavier load. The outbox matters just as much, because an agent-triggered write still has to propagate reliably. Workflow state matters more too, since the reasoning loop is exactly the kind of long-running, restart-surviving process that needs durable state and a correlation ID. And then the new one: conversation and context state. An agent carries context across turns, and that context has to live somewhere durable and queryable, which explains why a flexible document store keeps showing up as the default for agentic conversation state. The access pattern points at the store. Same principle as the map, applied to a new kind of state.

Governance and identity: where the assumptions break hardest

This layer changes most, and I’d insist any integration architect think it through before shipping an agentic workload.

The governance layer secured a deterministic platform. Identity answered who the caller was; policy constrained what the platform could be. Both still matter. However, agentic workloads open a gap that neither fully closes. Identity secures who the agent is. It does not touch what a poisoned tool result or a manipulated retrieved document makes the agent do. Prompt injection rides in through the data the agent requested inside the reasoning loop, downstream of the perimeter check everyone assumes protects them.

So the governance layer needs additions a deterministic platform never required:

Authorization moves per-action: A validated identity at the edge isn’t enough. Each tool call the agent makes needs its own check: is this specific action allowed for this tenant right now? The perimeter check happens once; the risk recurs on every call inside the loop.
Recovery means compensation, not retry: Agent actions have side effects across systems. A failed sequence three actions deep can’t restart from the top; it needs compensating actions to undo what already happened. That’s saga-style thinking, and you design it; it doesn’t emerge.
Containment has to be possible: When an agent misbehaves, you stop it fast, and at more than one layer. Layered containment, from a single configuration flip-up to a full block, turns “contain the agent” from an incident-call debate into a seconds-long operation.
Evaluation becomes a first-class layer: Operational observability tells you the agent is running. It doesn’t tell you the agent’s outputs are quietly degrading. Under the EU AI Act’s oversight and transparency duties, that stops being optional polish and becomes evidence you’re meeting an obligation.

The readiness lens, asked again

The design-to-operation post posed the question that decides go-live: not “is it built?” but “can we operate it safely, recoverably, auditably, and predictably?” Agentic workloads sharpen every word of that sentence.

Safely now includes per-action authorization and containment, because the threat walks in as data. Recoverably now means compensation and sagas, because retry alone can’t undo side effects. Auditably now covers what the agent accessed, which tool it called, why it acted, and what policy constrained it evidence the EU AI Act increasingly expects. And predictably is precisely the property the agent gave up, which is why the surrounding architecture has to supply it instead. The production baseline, the demonstrable-versus-designed test, the three moments of readiness all of it still applies. Each bar sits higher.

The revised framework

The map closes with five questions. For agentic workloads, they hold, and each gains a harder edge. Before an agentic workload goes near a real system, I’d add these:

Can I host a long-running reasoning loop, not just a request or an event? Can my orchestration stay correct when I can’t predict the action sequence? Does my data layer hold conversation state as well as business state, with idempotency doing heavier duty? Is authorisation per-action, not just per-identity? Can I contain a misbehaving agent in seconds? And can I evidence what the agent did, why, and whether its quality held?

Those aren’t different questions from the series. They’re the same layers, asked again under non-determinism, and then held up against the readiness lens one more time.

The shape of it

Agentic workloads don’t replace the Azure PaaS foundation an integration architect builds on. They stress it. Every layer in this series still includes compute, messaging, data, governance, and observability, but each one now supports a workload that decides for itself at runtime. The compute layer meets the loop. The messaging layer meets a non-deterministic router. The data layer meets conversation state and heavier idempotency. The governance layer meets a threat that walks in through the front door as data. And the readiness lens meets a workload that surrendered predictability, so the architecture has to supply it.

The through-line of the whole series holds here too. The model is the least differentiated part of a production agent. What separates a demo from something you can run against real systems in a regulated industry is the architecture around it: the same layers, asked harder, and proven in operation rather than promised in design. So the foundation was never wasted. It’s exactly what agentic workloads need, applied with the assumptions made explicit.

That’s the series. Start at the Azure PaaS map for the layer-by-layer foundation, take the design-to-operation lens with you as the test, and come back here for what changes when the workload thinks for itself.