Why Cosmos DB Ends Up as the Agent Memory Database

Posted on by

The first post in a series on Cosmos DB agent memory for AI agents, starting nine years before “AI agent” was a category.

In 2017, I built a proof of concept for a customer: a knowledge base on Cosmos DB, using the Graph model and Search, running at roughly 1,000 euros a month. I presented it at CloudBrew. One attendee wasn’t impressed:

“The most uninteresting talk of the day came from Steef-Jan Wiggers, who, in my opinion, delivered an hour-long marketing pitch for CosmosDB. I think it’s expensive for what it currently offers, and many developers could architect something with just as much performance without needing CosmosDB.”

He wasn’t wrong that 1,000 euros a month raises eyebrows as a line item. He was wrong about what the line item paid for: the knowledge base was the product a subscription business planned to sell. Compare the cost to the revenue it enabled, and it’s negligible. Compare it to nothing, and of course it looks “expensive.” I made the same point about Figma’s AWS bill last year: $109 million a year sounds alarming until you check it against $821 million in revenue and a business model that requires sub-100ms real-time collaboration for 13 million users. In short, cost without context is just a number that sounds big.

So here’s the same argument, nine years later, with a different workload. Agent memory, chat turns, tool call results, embeddings, and user preferences are expensive to store the wrong way and reasonably cheap to store the right way, and increasingly “the right way” means one database instead of three. I’ll come back to the actual RU numbers in post 6; for now, this post is about why the architecture argument holds up before cost even enters the picture.

The Same Shape of Problem, Nine Years Apart

Strip away “AI agent” and look at what you’re actually storing: short-lived, high-volume, time-ordered records that need fast writes and selective recall. That’s chat turns and tool outputs today. It’s also, structurally, what I modeled in a Cosmos DB Conf 2023 talk on end-to-end retail process monitoring, messages and batches flowing between an ERP, a WMS, and a PIM system, which I tracked so a retailer could tell where something broke.

Different domain, same shape, though: append-heavy writes, a need to reconstruct “what happened, in order,” and a downstream system (an incident manager then, an LLM now) that needs the right slice of history on demand, not the whole history every time.

In general, agent memory falls into two categories:

  • Short-term (episodic/working) memory — the last 5–10 turns of a conversation, intermediate tool call results, partial task state. Useful for the current task, disposable afterward (Cosmos DB’s time-to-live feature is a natural fit here; more on that in post 2).
  • Long-term memory — user preferences, summarized threads, facts the agent should persist and recall across sessions.

Both need somewhere to live, but the default answer for the last few years has been: somewhere different.

Why the Stitched Stack Breaks Down for Cosmos DB Agent Memory

The common pattern from 2022 through 2025 was to give each concern its own database: an in-memory store for caching and session state, a relational database for operational data and conversation logs, a purpose-built vector database for embeddings. A reasonable instinct, in theory: each tool for its own job.

In practice, though, it doesn’t hold up once an agent is the thing reading and writing across all three, on every turn.

Diagram comparing a stitched cache, SQL, and vector stack to a unified Cosmos DB agent memory layer feeding an AI agent
Three databases, three failure modes versus one Cosmos DB agent memory layer.

Each piece has a real weakness once agents are the workload, not an afterthought:

  • Pure vector databases tend to offer no strong read/write guarantees, limited ingestion throughput, availability below 99.9%, a single (eventual) consistency level, and thin multitenancy support. Fine for an embeddings side-project. Shaky as the record of what an agent told a customer.
  • Relational databases fight the fluid, evolving schema of agent state, new fields, new memory types, and nested tool outputs without migrations and, often, downtime.
  • In-memory caches are fast and don’t persist, which is exactly the opposite of what long-term memory needs.

As a result, three systems also means three consistency models, three availability profiles, and three places a multi-agent system can silently desynchronize. And that complexity tax doesn’t show up in any single service’s bill, which is part of why it’s easy to miss until something breaks in production.

The Unified Case — and How Much of It I’d Already Used

The pitch for Cosmos DB agent memory as a unified layer rests on a small set of properties: single-digit-millisecond latency, a 99.999% availability SLA on the NoSQL API, DiskANN-based vector indexing built into the same store as the operational data, multi-master writes, and five selectable consistency levels from strong to eventual. In plain terms, that’s one system that’s fast enough for the hot path, available enough for production, and flexible enough to hold embeddings next to the record they came from.

In fact, two of the pieces that make this work aren’t new to me, or new to this blog.

Timeline showing the author's Cosmos DB work from a 2017 proof of concept to 2023 retail monitoring to 2026 agent memory
Change feed and hierarchical partition keys weren’t built for AI agents; they just turned out to be exactly what Cosmos DB agent memory needed.

Change feed, for example. In the retail monitoring solution, change feed was the mechanism that turned a write into a trigger: a new record landing in Cosmos DB fired a Function, which could raise an incident. That’s the same primitive I’ll use in post 4 to coordinate handoffs between agents in a multi-agent system: one agent’s write becomes another agent’s signal to act, without polling.

Hierarchical partition keys, likewise. I covered these in Azure Cosmos DB’s Latest Performance Features back in 2023: partitioning by tenant, then by item, to keep related data colocated while avoiding the 20 GB logical partition ceiling. The mechanism hasn’t changed; what’s changed is the workload. Post 2 uses the same [tenantId, threadId] pattern to isolate one customer’s agent conversations from another’s.

Even so, I didn’t build either feature for AI agents. Both turned out to be exactly what agent memory needs a decent sign that the underlying database was solid before the AI use case arrived, and nobody retrofitted it to fit.

Where This Series Is Headed

This post is the framing argument for Cosmos DB agent memory. From here, the rest of the series gets specific:

  • Post 2 — designing the agent memory schema itself: partition key choice, TTL, and the turn-based data model that works best in practice.
  • Post 3 — vector, full-text, and hybrid search for recalling the right memories, not just any memories.
  • Post 4 — multi-agent state and coordination, including change feed as the handoff mechanism.
  • Post 5 — wiring Cosmos DB into Microsoft Foundry Agent Service as bring-your-own thread storage.
  • Post 6 — the cost conversation, properly this time: RU drivers, semantic caching, and what this actually costs to run at scale.

The 2017 knowledge base cost 1,000 euros a month and paid for itself many times over as a revenue-generating product. Ultimately, the question worth asking about agent memory infrastructure in 2026 isn’t “is this expensive”; it’s the same question it always was: expensive relative to what?

Sources

Leave a Reply