Why Cosmos DB Ends Up as the Agent Memory Database

Posted on August 30, 2026 by steefjan1970

The first post in a series on Cosmos DB agent memory for AI agents, starting nine years before “AI agent” was a category.

In 2017, I built a proof of concept for a customer: a knowledge base on Cosmos DB, using the Graph model and Search, running at roughly 1,000 euros a month. I presented it at CloudBrew. One attendee wasn’t impressed:

“The most uninteresting talk of the day came from Steef-Jan Wiggers, who, in my opinion, delivered an hour-long marketing pitch for CosmosDB. I think it’s expensive for what it currently offers, and many developers could architect something with just as much performance without needing CosmosDB.”

He wasn’t wrong that 1,000 euros a month raises eyebrows as a line item. He was wrong about what the line item paid for: the knowledge base was the product a subscription business planned to sell. Compare the cost to the revenue it enabled, and it’s negligible. Compare it to nothing, and of course it looks “expensive.” I made the same point about Figma’s AWS bill last year: $109 million a year sounds alarming until you check it against $821 million in revenue and a business model that requires sub-100ms real-time collaboration for 13 million users. In short, cost without context is just a number that sounds big.

So here’s the same argument, nine years later, with a different workload. Agent memory, chat turns, tool call results, embeddings, and user preferences are expensive to store the wrong way and reasonably cheap to store the right way, and increasingly “the right way” means one database instead of three. I’ll come back to the actual RU numbers in post 6; for now, this post is about why the architecture argument holds up before cost even enters the picture.

The Same Shape of Problem, Nine Years Apart

Strip away “AI agent” and look at what you’re actually storing: short-lived, high-volume, time-ordered records that need fast writes and selective recall. That’s chat turns and tool outputs today. It’s also, structurally, what I modeled in a Cosmos DB Conf 2023 talk on end-to-end retail process monitoring, messages and batches flowing between an ERP, a WMS, and a PIM system, which I tracked so a retailer could tell where something broke.

Different domain, same shape, though: append-heavy writes, a need to reconstruct “what happened, in order,” and a downstream system (an incident manager then, an LLM now) that needs the right slice of history on demand, not the whole history every time.

In general, agent memory falls into two categories:

Short-term (episodic/working) memory — the last 5–10 turns of a conversation, intermediate tool call results, partial task state. Useful for the current task, disposable afterward (Cosmos DB’s time-to-live feature is a natural fit here; more on that in post 2).
Long-term memory — user preferences, summarized threads, facts the agent should persist and recall across sessions.

Both need somewhere to live, but the default answer for the last few years has been: somewhere different.

Why the Stitched Stack Breaks Down for Cosmos DB Agent Memory

The common pattern from 2022 through 2025 was to give each concern its own database: an in-memory store for caching and session state, a relational database for operational data and conversation logs, a purpose-built vector database for embeddings. A reasonable instinct, in theory: each tool for its own job.

In practice, though, it doesn’t hold up once an agent is the thing reading and writing across all three, on every turn.

Each piece has a real weakness once agents are the workload, not an afterthought:

Pure vector databases tend to offer no strong read/write guarantees, limited ingestion throughput, availability below 99.9%, a single (eventual) consistency level, and thin multitenancy support. Fine for an embeddings side-project. Shaky as the record of what an agent told a customer.
Relational databases fight the fluid, evolving schema of agent state, new fields, new memory types, and nested tool outputs without migrations and, often, downtime.
In-memory caches are fast and don’t persist, which is exactly the opposite of what long-term memory needs.

As a result, three systems also means three consistency models, three availability profiles, and three places a multi-agent system can silently desynchronize. And that complexity tax doesn’t show up in any single service’s bill, which is part of why it’s easy to miss until something breaks in production.

The Unified Case — and How Much of It I’d Already Used

The pitch for Cosmos DB agent memory as a unified layer rests on a small set of properties: single-digit-millisecond latency, a 99.999% availability SLA on the NoSQL API, DiskANN-based vector indexing built into the same store as the operational data, multi-master writes, and five selectable consistency levels from strong to eventual. In plain terms, that’s one system that’s fast enough for the hot path, available enough for production, and flexible enough to hold embeddings next to the record they came from.

In fact, two of the pieces that make this work aren’t new to me, or new to this blog.

Change feed, for example. In the retail monitoring solution, change feed was the mechanism that turned a write into a trigger: a new record landing in Cosmos DB fired a Function, which could raise an incident. That’s the same primitive I’ll use in post 4 to coordinate handoffs between agents in a multi-agent system: one agent’s write becomes another agent’s signal to act, without polling.

Hierarchical partition keys, likewise. I covered these in Azure Cosmos DB’s Latest Performance Features back in 2023: partitioning by tenant, then by item, to keep related data colocated while avoiding the 20 GB logical partition ceiling. The mechanism hasn’t changed; what’s changed is the workload. Post 2 uses the same [tenantId, threadId] pattern to isolate one customer’s agent conversations from another’s.

Even so, I didn’t build either feature for AI agents. Both turned out to be exactly what agent memory needs a decent sign that the underlying database was solid before the AI use case arrived, and nobody retrofitted it to fit.

Where This Series Is Headed

This post is the framing argument for Cosmos DB agent memory. From here, the rest of the series gets specific:

Post 2 — designing the agent memory schema itself: partition key choice, TTL, and the turn-based data model that works best in practice.
Post 3 — vector, full-text, and hybrid search for recalling the right memories, not just any memories.
Post 4 — multi-agent state and coordination, including change feed as the handoff mechanism.
Post 5 — wiring Cosmos DB into Microsoft Foundry Agent Service as bring-your-own thread storage.
Post 6 — the cost conversation, properly this time: RU drivers, semantic caching, and what this actually costs to run at scale.

The 2017 knowledge base cost 1,000 euros a month and paid for itself many times over as a revenue-generating product. Ultimately, the question worth asking about agent memory infrastructure in 2026 isn’t “is this expensive”; it’s the same question it always was: expensive relative to what?

Sources

Microsoft Learn — AI agents in Azure Cosmos DB
Microsoft Learn — Time to live (TTL) in Azure Cosmos DB.
Your own archive — Azure Cosmos DB’s Latest Performance Features
Your own archive — Figma AWS Costs Explained: Beyond the Hype and Panic
Microsoft Learn Shows — Leveraging Azure Cosmos DB for End-to-End Monitoring of Retail Processes

Azure Observability and FinOps for Integration Architects

Posted on August 20, 2026 by steefjan1970

In the Azure PaaS map post, observability was folded into the governance layer, with a note that Application Insights and Azure Monitor are non-negotiable. That was true, but it undersold the topic. For an integration platform specifically, observability isn’t a sub-bullet of governance. It’s the layer that decides whether you can actually run the thing in production.

So this post pulls observability out and gives it room. And it brings FinOps along, because the two share a root: you can’t manage what you can’t see. One makes system behaviour visible; the other makes cost visible. Both turn a platform from “it runs” into “we can run it responsibly.” Azure observability and FinOps, treated together, are what separate a platform that works in a demo from one you can operate under real load.

The gap between design and demonstrable operation

Here’s the pattern I see most often on integration platforms. The observability design is excellent. There’s a logging standard, a tracing approach, a set of required fields. Then you look at the actual infrastructure, and none of it is enforced. The alert rules aren’t there. The dashboards aren’t built. The diagnostic settings aren’t wired. The design lives in a document; the platform doesn’t know about it.

That gap matters more than it sounds. A monitoring standard that depends on discipline and review isn’t a platform capability; it’s a hope. The moment a team ships an integration without the dashboards, the standard quietly failed. So the real work in this layer isn’t designing observability. It’s making observability demonstrable: wired into the infrastructure, enforced in the pipeline, and impossible to skip.

Let’s walk what that means in practice.

OpenTelemetry as a platform contract, not a suggestion

Most mature integration platforms land on OpenTelemetry as the instrumentation standard. That’s the right call. W3C Trace Context propagates a trace across services, traces and metrics and logs share a model, and you avoid inventing your own correlation scheme. So far, so good.

The catch is that “we use OpenTelemetry” is a design statement, not an enforced one. For it to be a contract, three things must be true. First, the required fields, resource attributes, trace fields, and domain identifiers have to be defined explicitly, not left to each team’s judgment. Second, that definition has to be validated somewhere automatically, ideally at pull request. Third, the platform components themselves have to emit the standard, so a trace actually runs unbroken from the API gateway through messaging to the backend. Miss any of those, and you have telemetry that mostly correlates, which is worse than none, because it looks trustworthy right up until the incident where it isn’t.

Tracing the chain, not just the components

Azure gives you per-resource monitoring for free. You can see API Management’s metrics, Service Bus’s queue depth, and a Function’s execution count. That’s component monitoring, and it’s necessary but not sufficient. An integration platform’s job is to move a message across those components, so the question that matters is whether you can follow a single message or transaction through the entire chain.

That end-to-end view has to map onto the layers of your integration architecture, because each layer asks a different question. The consumer-facing layer cares about availability, latency, error rates, and throttling per channel. The process layer cares about routing, transformations, retries, and failures in async steps. The system-facing layer cares about dependencies on backends’ response times, timeouts, and contract breaks. Without that layered, chain-aware view, you get plenty of technical detail per Azure resource and almost no ability to reason about the integration as a whole.

Message-level insight and the async recovery problem

Component metrics tell you the platform is busy. They don’t help the person who has to answer “what happened to order 47821?” For that, an operator needs message-level insight: business identifiers, error categories, chain status, the last successful step, and retry state. Structured logging with domain attributes a flow ID, a message ID, a route key, and an error category is what makes that possible. And it has to come with explicit data classification, masking, retention, and access rules, because business identifiers in logs are exactly the kind of data a regulator asks about.

Then there’s recovery, which is where the compute choice comes back to bite. Async, message-driven processing needs a replay story: when something fails partway through, you need to know how far it got and re-drive it from there. A workflow engine often gives you some of this out of the box. Raw compute like Functions doesn’t, so you have to design the replay mechanism yourself, as part of the integration pattern rather than an afterthought.

The pattern that works: treat the message on the bus as a reference, not the full payload. Pair it with the claim-check pattern, in which the bus carries technical and functional metadata: trace ID, flow ID, message ID, route key, error category, retry count, and a pointer to the payload, safely stored in storage. Define checkpoints along the flow. Then, on failure, you can determine where processing succeeded and re-drive from the right point, with idempotency (from the data patterns post) making the re-drive safe. For fully synchronous request-response, re-driving belongs with the caller; the platform’s job there is clear error codes and traceability.

Monitoring as a Definition of Done

The single highest-leverage move in this layer costs almost nothing: make monitoring a Definition of Done for every integration. No integration ships without its dashboard, its alerts, its trace-context propagation, its required log fields, its retention setting. And this is the part that turns it from aspiration into capability: the checklist runs as a quality gate in the pipeline, not as a line in a review someone might skip.

That one change moves observability from “depends on the discipline of whoever built it” to “the platform won’t let you skip it.” It’s the difference between a standard and an enforced standard, and it’s the cheapest high-value thing on this entire list.

FinOps: cost is just another signal you can’t yet see

Everything above is about making system behavior visible. FinOps is the same discipline applied to cost. On an integration platform, it fails in the same way because cost visibility typically ends at the subscription or resource group boundary. That’s too coarse. It can’t tell you what an individual integration costs, or an API, or a queue, or a team’s share of a shared component.

Three FinOps problems come up on every integration platform:

Attribution needs a taxonomy. Without a consistent tagging scheme for value stream, team, environment, integration, API, owner, and cost category, cost remains a lump sum. With one, you can steer on cost per integration product rather than cost per subscription. This is the foundation; nothing else works without it.
Shared components are the hard part. Compute is easy to attribute when each team runs its own. But a shared API Management instance, a shared Service Bus namespace, a shared Log Analytics workspace those get used by everyone and billed centrally, and if you never build a distribution model, nobody owns the cost. The shared components that make the platform economical are exactly the ones whose cost is hardest to place. That’s not a reason to isolate everything; it’s a reason to deliberately decide the split.
Storage and retention are FinOps levers hiding within a compliance requirement. Observability generates data logs, traces, payloads held for replay, and dead-lettered messages. Compliance dictates how long you keep it. But retention length and storage tier are separate decisions. Data you must keep for audit doesn’t have to sit in a hot, queryable tier the whole time. Tie retention to data classification, then move cold data to cheaper tiers. The requirement is “keep it”; the FinOps move is “keep it cheaply.”

The through-line: FinOps on an integration platform isn’t financial reporting after the fact. It’s a design and governance concern, sitting right next to observability, because both are about seeing what the platform is actually doing.

Where this layer gets over-applied

Consistent with the series, the honesty section. Observability and cost control both have a failure mode of doing too much.

Not every signal deserves an alert. An alert that fires on something nobody acts on trains people to ignore alerts. Alert on what changes a decision; leave the rest on a dashboard. Alert fatigue is a real operational risk, not a sign of thoroughness.

Not every message needs full payload logging. Metadata-first is the right default. Payload logging belongs where there’s functional need and explicit consent, with masking and retention — not everywhere, because “log everything” is how sensitive data ends up somewhere it shouldn’t, and how your storage bill quietly triples.

Not every cost needs fine-grained attribution. Building per-message cost tracking for a low-volume internal integration spends more effort than the insight is worth. Match the granularity of attribution to the scale of the spend.

The shape of it

For an integration architect, observability and FinOps answer the same question in two currencies: what is the platform actually doing, and what is it actually costing? Wire OpenTelemetry in as an enforced contract. Trace the chain, not just the components. Give operators message-level insight and a real replay story. Make monitoring a Definition of Done the pipeline enforces. Then apply the same visibility to cost: a tagging taxonomy, a distribution model for shared components, and retention tiered by classification. Get both right, and the platform stops being a black box you hope is behaving and becomes one you can actually operate.

Want the layer this sits inside? The Azure PaaS map puts observability and governance in context against compute, integration, and data, and walks the five-question framework across all of them.

Token Economics in Practice: What the Citadel Cost Attribution Policy Actually Meters

Posted on August 4, 2026 by steefjan1970

The FinOps Foundation published a piece called Token Economics: The Atomic Unit of AI Value, and it’s one of the better attempts I’ve seen at giving AI cost management a real vocabulary. Tokens as the atomic unit of cost, goodput instead of raw throughput, and a warning that the token meter is increasingly hidden inside SaaS subscriptions you don’t control.

Most writing on this topic stays theoretical. I have something to test it against. The Citadel Platform series on this blog built cost attribution and semantic caching into a real APIM gateway, running in Sweden Central, metering a real agent. So instead of summarizing the FinOps article, this post uses it as a checklist. Where does the Citadel implementation already deliver on token economics, and where does it fall short?

The honest answer: it holds up well on attribution and caching, and it has clear gaps on goodput, yield, and routing. Let’s go through it.

Three token economics ideas worth carrying forward

I’ll paraphrase the three concepts I’m testing against, and you should read the original for the full argument.

Goodput, not throughput. Raw token volume tells you what you spent, not what you got. Goodput asks how many of those tokens produced useful output within acceptable latency. A retry storm and a productive session can burn the same token count.

The cost stack extends beyond the token. Tokens are the atomic unit, but the bill includes orchestration overhead, retries, tool-calling scaffolding, and increasingly, SaaS subscriptions that embed token consumption behind a flat price. When the meter sits inside someone else’s product, you lose the visibility FinOps depends on.

Engineering levers matter more than procurement levers. Model routing, semantic caching, and compressing tool-calling overhead move the cost curve more than negotiating a discount does. The FinOps article cites Cloudflare’s Code Mode work, which cut MCP tool-schema token overhead dramatically by changing how tools present themselves to the model.

Now let’s hold the Citadel Hub against those three ideas.

What the Citadel Hub already meters

Every call the weather agent makes flows through apim-wpvlimv4ngkns, and two of the five governance policies from earlier in the series do the token economics work.

The cost attribution policy emits token metrics per call, dimensioned by subscription and agent:

xml

			
<azure-openai-emit-token-metric namespace="citadel">
    <dimension name="Subscription ID" />
    <dimension name="Agent ID" value="@(context.Request.Headers.GetValueOrDefault("X-Agent-Id", "unknown"))" />
    <dimension name="API ID" />
</azure-openai-emit-token-metric>

		

That gives us prompt tokens, completion tokens, and total tokens per agent, per subscription, queryable in Application Insights. When someone asks what the weather agent cost last week, the answer is a query, not an estimate.

The semantic caching policy sits in front of the model and short-circuits repeat questions:

xml

			
<azure-openai-semantic-cache-lookup
    score-threshold="0.85"
    embeddings-backend-id="embeddings-backend"
    embeddings-backend-auth="system-assigned">
    <vary-by>@(context.Request.Headers.GetValueOrDefault("X-Agent-Id", "unknown"))</vary-by>
</azure-openai-semantic-cache-lookup>

		

A cache hit costs an embedding call instead of a full completion. For an agent that answers weather questions, where “what’s the weather in Amsterdam” arrives in twenty phrasings, that’s not a rounding error.

In FinOps for AI terms, the first policy lives in the Understand Usage and Cost domain, and the second in Optimize Usage and Cost. So far, the framework and the implementation agree.

Where the mapping holds up

Two places, and one of them matters more than I expected before reading the article.

Semantic caching is a named lever. The FinOps article lists it explicitly as an engineering-side optimization, and the Citadel implementation has it running in production policy XML, not on a roadmap slide. Score threshold tuning is real work (0.85 took iterations, and I documented the false-positive risk in the original policy deep dive), but the lever exists, and it’s been pulled.

The gateway is the anti-aggregator, and the FinOps article’s sharpest warning is that token consumption is being hidden in SaaS subscriptions, where a flat monthly price hides a metered reality beneath the surface. The hub-and-spoke model is the architectural inverse of that problem. Nothing reaches a model without crossing the gateway, so nothing consumes tokens invisibly. The whole point of Part 2 in the Citadel series was to refuse the path that bypasses the meter when the Agent Service SDK tries to call the model directly.

I’d go one step further than the FinOps article does. Centralized metering isn’t just a FinOps convenience. It’s the same choke point that enforces content safety and the kill switch. Cost visibility and governance aren’t two systems in this architecture, they’re one policy pipeline.

Where Citadel’s token economics fall short

This is the useful part, because the gaps are specific.

No goodput tracking: The Hub knows how many tokens the agent consumed. It does not know how many of them were worth consuming. Time-to-first-token and tokens-per-second aren’t captured as dimensions, and nothing distinguishes a completion the user acted on from one that got regenerated three times. By the article’s standard, Citadel measures throughput and calls it a day.
No token yield rate: Closely related, but distinct. Yield asks: cost per successful outcome, not per call. The weather agent writes every conversation to Cosmos DB (Part 3 of the series), so the raw material for outcome tagging exists. Nothing joins it to the token metrics yet. That’s a gap in instrumentation, not in data.
No model routing: Every query hits the same deployment, whether it’s “weather in Ede” or a multi-step tool-calling chain. The article’s Pareto framing (bulk tokens, mid-tier tokens, premium low-latency tokens, reasoning tokens, drawn from SemiAnalysis’s InferenceX benchmarking) implies a cascade: cheap model first, escalate on need. APIM can express this with backend pools and routing policy. Citadel doesn’t, yet.
Tool-schema overhead is unmeasured: Every tool-calling request carries the Open-Meteo tool definition in the payload, on every single call. One tool, so the overhead is small. But the Cloudflare finding the article cites is a warning about what happens at ten or twenty tools, and I have no metric today that would even show me the problem growing.

Why this bites harder on agentic workloads

There’s a compounding effect the article touches on that I can back with a documented example. Orchestration overhead isn’t a fixed tax, it multiplies through agent chains.

In the Logic Apps Agent Loop series, I found that sequential agents don’t pass plain strings between each other. Each agent action returns a structured JSON messages array, and you need a Compose action to bridge it into the next agent. Every one of those bridged payloads is tokens. Single-agent token math is linear. Multi-agent token math is not, and that’s where token economics stops being a dashboard exercise. If your metering only captures totals per call, the orchestration overhead hides inside numbers that look individually reasonable.

What I’d add to the Citadel Hub next

In order of effort against payoff:

Outcome tagging first: The conversations container already holds every run. Adding a resolution field (answered, retried, abandoned) and joining it against the token metrics in Application Insights gets me a real token yield rate with no new infrastructure. This is the cheapest gap to close and the one that changes the conversation from “what did we spend” to “what did we get.”
Latency dimensions second: Emitting time-to-first-token and total duration alongside the existing token dimensions turns the same App Insights workspace into a goodput dashboard. APIM sees the timing already, it just doesn’t emit it.
A routing experiment third: The weather agent is a good candidate for a two-tier cascade precisely because it’s boring. Simple lookups go to a small model, tool-calling chains escalate. If the cascade breaks the agent, it breaks it cheaply, and I’ll write up whatever goes wrong.

Tool-schema compression stays on the watch list rather than the to-do list. With one tool, measuring it first beats optimizing it blind.

Pitfalls

Adopting the vocabulary without the substance is the most common trap. It’s easy to say ‘we do token economics’ because a dashboard shows token counts. Raw volume without yield or goodput is accounting, not economics. The article’s framework is only useful if the uncomfortable metrics come with it.

Treating flat-price AI tools as flat costs is the second trap. When teams around you adopt AI SaaS tooling, those subscriptions consume tokens on someone’s meter. Budgeting them as fixed line items repeats the exact mistake the article warns about, one procurement layer up.

Optimizing the cache before understanding the traffic is the last one. A semantic cache with an aggressive threshold saves tokens and quietly serves wrong answers. Tune against logged real queries, never against the token savings number alone. I learned this at 0.85, and the number that’s right for a weather agent is wrong for an agent where two similar-sounding questions need different answers.

Closing

The FinOps article gives this space the vocabulary it needs, and the Citadel Platform gives me somewhere to test that vocabulary against running policy XML. The scorecard: attribution and caching, solid. Goodput, yield, and routing: real gaps with concrete next steps.

The bigger takeaway is architectural. Every improvement on that list lands in the same place, the gateway. APIM started this series as a governance layer. It’s ending it as the FinOps instrumentation layer too, and I don’t think that’s a coincidence. The choke point that can say no to a request is the same choke point that can tell you what the request cost.

If you’re metering your own agent platform, I’d like to hear which of these gaps you closed first, and whether the yield numbers surprised you.

Azure API Management Token Metric Policy: AI Cost Observability and Cross-Charging

Posted on May 20, 2026 by steefjan1970

Part 4 of 7 in the “APIM for AI Workloads” series

The Azure API Management token metric policy turns AI cost data from a finance problem into an engineering one. In Part 3, we covered enforcement: how to set consumption boundaries per consumer. This post covers the complementary piece: how to measure that consumption. More importantly, it shows how to make it visible to the right people and use it to drive internal cross-charging and FinOps dashboards.

At my current company, one of the first questions the architecture board asked was straightforward: which teams are consuming what, and what does it cost? Without instrumentation at the gateway layer, that question is genuinely unanswerable. The token metric policy is how you answer it.

Azure API Management Token Metric Policy: How It Works

The policy sits in the outbound section of your APIM pipeline. After the AI backend returns a response, APIM reads the token usage fields from the response body. These include prompt tokens, completion tokens, and total tokens. APIM then emits them as custom metrics to Application Insights under a namespace you define.

Crucially, the policy emits metrics after the response arrives. It uses actual token counts from the API response rather than estimates. As a result, the data is accurate rather than approximated. It also means the metric emission adds no latency to the request path: the response is returned to the caller immediately, and the metric is emitted asynchronously.

The generic variant, llm-emit-token-metric, works identically for non-Azure backends. Both policies share the same dimension model, so the configuration patterns below apply regardless of which AI provider sits behind APIM.

Choosing Dimensions for Azure API Management Token Metric Policy

Dimensions are the labels attached to each metric event. They explain how to slice and aggregate token consumption data in Application Insights. Choosing the right dimensions is the most important configuration decision for making the data useful for cross-charging.

The three primary dimension options are:

Subscription ID. The most common choice for internal enterprise deployments. Each APIM subscription maps to a team, product, or cost center, so filtering Application Insights metrics by Subscription ID gives you direct per-team token consumption. This pairs naturally with the subscription key authentication pattern from Part 2 and the per-subscription counter-key from Part 3.

User ID. Sourced from the JWT subject claim or a custom header, User ID enables per-user consumption reporting. This is the right dimension for multi-tenant SaaS applications where individual end users have their own token budgets, or where you need to identify heavy consumers within a shared subscription.

API ID. Identifies which APIM API product generated the consumption. Useful when a single subscription uses multiple AI-backed APIs: one for a conversational agent, one for content generation, and one for document summarization. API ID lets you break down cost by use case rather than just by subscriber.

In practice, combining all three dimensions gives you the most flexibility. A single metric event tagged with Subscription ID, User ID, and API ID can answer questions at every level: how much did the platform spend in total, how much did Team A spend, how much did User X consume, and which AI feature is the most expensive to run.

Querying Token Metrics in Application Insights

Once the policy is emitting metrics, you query them in Application Insights using the custom metrics namespace you configured. The metrics appear under the namespace name you set in the policy (for example, “AzureOpenAI” or “MyLLM”), with separate metric events for prompt tokens and completion tokens.

A practical starting point is a KQL query that aggregates the total number of tokens by Subscription ID over the past 30 days. From there, you can add filters by API ID to isolate specific workloads, or pivot by User ID to identify the highest consumers within a team.

For FinOps dashboards, the most useful view is a stacked time-series chart of total token consumption broken down by subscription, updated daily. This gives finance and engineering a shared view of AI spend trends without exporting data from Azure Monitor to a separate BI tool. Azure Workbooks can host this directly in the Azure portal, making it accessible to non-technical stakeholders.

From Observability to Cross-Charging

Observability is the prerequisite for cross-charging. However, they are not the same thing. Observability tells you what happened. Cross-charging, by contrast, is the organizational process of allocating those costs to the right budget owners.

The token metric policy gives you the raw data. To turn that into a cross-charge, you need two additional steps. First, agree on a price per token with your finance team — usually derived from the Azure cost per 1,000 tokens for your model and region. Second, automate a monthly report that multiplies token consumption by the subscription price.

This does not need to be complex. For example, a Logic App or Azure Function that queries Application Insights on the first of each month works well for most organizations starting out. It aggregates tokens by subscription, multiplies by the agreed rate, and emails a cost summary to each team lead. The Application Insights REST API makes this straightforward to automate.

Finally, the most important advice: have this conversation with finance and product teams before AI consumption scales. Retroactive cross-charging is significantly harder to establish than an upfront model with clear methodology and tooling.

What’s Next in This Azure API Management for AI Series

Part 5 covers load balancing and circuit breaking: how to distribute traffic across PTU and PAYG backends, configure backend pools, and set up circuit breaker rules for automatic failover when a primary endpoint becomes unavailable.

Part 5: Load balancing and circuit breaking across PTU and PAYG backends.
Part 6: Semantic caching — reducing token consumption with similarity-based response reuse.
Part 7: APIM as an MCP gateway for agentic AI workloads.

Figma AWS Costs Explained: Beyond the Hype and Panic

Posted on July 16, 2025 by steefjan1970

Figma’s recent IPO filing revealed that its Figma AWS costs amount to roughly $300,000 per day, approximately $109 million annually, or 12% of its reported revenue of $821 million. The company is also committed to a minimum spend of $545 million with AWS over the next five years. Cue the online meltdown. “Figma is doomed!” “Fire the CTO!” The internet, in its infinite wisdom, declared. I wrote a news item on it for InfoQ and thought, let’s put things into perspective.

(Source: Figma.com)

But let’s inject a dose of reality, shall we? As Corey Quinn from The Duckbill Group, who probably sees more AWS invoices than you’ve seen Marvel movies, rightly points out, this kind of spending for a company like Figma is boringly normal.

As Quinn extensively details in his blog post, Figma isn’t running a simple blog. It’s a compute-intensive, real-time collaborative platform serving 13 million monthly active users and 450,000 paying customers. It renders complex designs with sub-100ms latency. This isn’t just about spinning up a few virtual machines; it’s about providing a seamless, high-performance experience on a global scale.

The Numbers Game: What the Armchair Experts Missed About Figma AWS Costs

The initial panic conveniently ignored a few crucial realities, according to Quinn:

Ramping Spend: Most large AWS contracts increase year-over-year. A $109 million annual average over five years likely starts lower (e.g., $80 million) and gradually increases to a higher figure (e.g., $150 million in year five) as the company expands.
Post-Discount Figures: These spend targets are post-discount. At Figma’s scale, they’re likely getting a significant discount (think 30% effective discount) on their cloud spend. So, their “retail” spend would be closer to $785 million over five years, not $545 million.

When you factor these in, Figma AWS costs fall squarely within industry benchmarks for its type of business:

Compute-lite SaaS: around 5% of revenue
Compute-heavy platforms (like Figma): 10–15% of revenue
AI/ML-intensive companies: often exceeding 15%

At 12% of revenue, Figma’s AWS costs are exactly where you’d expect them to be for a platform delivering real-time collaborative experiences at a global scale.

Furthermore, the increasing adoption of AI and Machine Learning in application development is introducing a new dimension to cloud costs. AI workloads, particularly for training and continuous inference, are incredibly resource-intensive, pushing the boundaries of compute, storage, and specialized hardware (like GPUs), which naturally translates to higher cloud bills. This makes effective FinOps and cost optimization strategies even more crucial for companies that leverage AI at scale.

So, while the internet was busy getting its math wrong and forecasting doom, Figma was operating within a completely reasonable range for its business model and scale.

The “Risky Dependency” Non-Story

Another popular narrative was the “risky dependency” on AWS. Figma’s S-1 filing includes standard boilerplate language about vendor dependencies, a common feature found in virtually every cloud-dependent company’s SEC filings. It’s the legal equivalent of saying, “If the sky falls, our business might be affected.”

Breaking news: a SaaS company that uses a cloud provider might be affected by outages. In related news, restaurants depend on food suppliers. This isn’t groundbreaking insight; it’s just common business risk disclosure. Figma’s “deep entanglement” with AWS, as described by Hacker News commenter nevon, illustrates the complexity of modern cloud architectures. Every aspect, from permissions to disaster recovery, is seamlessly integrated. That makes a quick migration akin to open-heart surgery. Not something you do on a whim.

Cloud Repatriation: A Valid Strategy, But Not a Universal Panacea

Figma’s costs reignited the cloud repatriation debate. The most vocal advocate is 37signals CTO David Heinemeier Hansson, who famously exited the cloud to save millions. And he’s not wrong for some companies; repatriating workloads delivers significant savings. But it’s not a one-size-fits-all solution.

Every company’s needs are different. Scrimba, for example, runs on dedicated servers and spends less than 1% of revenue on infrastructure. For them, repatriation is a perfect fit. Figma is a different story. Its real-time collaborative demands and massive user base require agility, scalability, and managed services at a global scale. A hyperscale provider like AWS isn’t optional; it’s central to the business model.

This brings us to a broader conversation, especially relevant in Europe: digital sovereignty. As I’ve discussed in my blog post, “Digital Destiny: Navigating Europe’s Sovereignty Challenge,” deep integration with a single hyperscaler isn’t just a cost question. It also affects the control an organization retains over its data and operations. Vendor lock-in carries real strategic implications. Data governance, regulatory compliance, and negotiating power can all be compromised. The extraterritorial reach of foreign laws adds another layer of concern. Many organizations are responding by exploring multi-cloud strategies or hybrid models. The goal: mitigate risk and assert greater control over their digital destiny.

My Cloud Anecdote: Costs vs. Value

This whole debate reminds me of a scenario I encountered back in 2017. I was working on a proof of concept for a customer, building a future-proof knowledge base using Cosmos DB, the Graph Model, and Search. The operating cost, primarily driven by Cosmos DB, was approximately 1,000 euros per month. Some developers immediately flagged it as “too expensive,” as I can recall, or even thought I was selling Cosmos DB. The reception, however, wasn’t universally positive. In fact, one attendee later wrote in their blog:

The most uninteresting talk of the day came from Steef-Jan Wiggers, who, in my opinion, delivered an hour-long marketing pitch for CosmosDB. I think it’s expensive for what it currently offers, and many developers could architect something with just as much performance without needing CosmosDB.

However, the proposed solution was for a knowledge base that customers could leverage via a subscription model. The crucial point was that the costs were negligible compared to the potential revenue the subscription model would net for the customer. It was an investment in a revenue-generating asset, not just a pure expense.

The Bottom Line: Putting Figma AWS Costs in Perspective

Thanks to Quinn, I understand that Figma is actively optimizing its infrastructure, transitioning from Ruby to C++ pipelines, migrating workloads, and implementing dynamic cluster scaling. He concluded:

They’re doing the work. More importantly, they’re growing at 46% year-over-year with a 91% gross margin. If you’re losing sleep over their AWS bill while they’re printing money like this, you might need to reconsider your priorities.

The “innovation <-> optimization continuum” is always at play. Companies often prioritize rapid innovation and speed to market, leveraging the cloud for its agility and flexibility. As they scale, they can then focus on optimizing those costs, and Figma AWS costs are no exception to that pattern.

This increasing complexity underscores the growing importance of FinOps (Cloud Financial Operations), a cultural practice that brings financial accountability to the variable-spend model of cloud computing, empowering teams to make data-driven decisions about cloud usage and optimize costs without sacrificing innovation.

Figma’s transparency in disclosing its cloud costs is actually a good thing. It forces a much-needed conversation about the true cost of running enterprise-scale infrastructure in 2025. The hyperbolic reactions, however, expose a fundamental misunderstanding of these realities. Which I also encountered with my Cosmos DB project in 2017.

So, the next time someone tells you that a company spending 12% of its revenue on infrastructure that literally runs its entire business is “doomed,” perhaps ask them how much they think it should cost to serve real-time collaborative experiences to 13 million users across the globe. When you understand what drives Figma AWS costs, the answer might surprise you.

Lastly, as the cloud landscape continues to evolve, with new services, AI integration, and shifting geopolitical considerations, the core lesson remains: smart cloud investment isn’t about avoiding the bill, but understanding its true value in driving business outcomes and strategic advantage. The dialogue about cloud costs is far from over, but it’s time we grounded it in reality.

Should developers care about Azure Cost?

Posted on February 14, 2022 by steefjan1970

The days of prepurchasing a large amount of infrastructure are gone. Instead, in the Cloud, we deal with buying small units of resources at a low cost. As a result, developers have the freedom to provision resources and deploy their apps. They can spend company money at a click of a button or line of code. There is no longer a need to go through any procurement process.

Therefore you could ask the question: Should developers be aware of the running costs of their apps and belonging infrastructure? And also worry about SKU’s, dimensioning, and unattended resources? I would say yes, they should be aware. Depending on requirements, environments (dev, test, acceptance, and production), availability, security, test strategy, and so on, costs will accumulate. Having an eye on the cost from the start will prevent discussion when the bill is too high at the end of the month or lacks justifying of the chosen deployment of Azure resources.

Fortunately, there are services and tools available to help you in the estimation of costs, monitoring, and analysis for cost optimization. Furthermore, you can help identify costs by applying tags to your Azure resources – important when costs of Azure resources in a subscription are shared over departments.

Azure Calculator

Microsoft provides a Cloud Platform called Azure containing over 100 services for its customers. They are charged for most of the services when consuming them. These charges (cost) can be estimated using the so-called ‘Pricing calculator.’

You can search for a product (service) with the pricing calculator and subsequently select it.

Next, a pop window on the right-hand side will appear, and you click on view. Finally, a window will appear with the options for, in this case, Logic Apps. You can select the region where you like to provision your product (service), and depending on hosting, other criteria specify what you like to consume. In addition, you can select what type of support you want and licensing model – and there is also a switch allowing you to see what the dev/test pricing is for the product.

Furthermore, if you want to estimate a solution consisting of multiple products, you can select all of them before specifying the consumption characteristics. The calculator will, in the end, show the accumulated costs for all products.

Other tabs in the calculator showcase sample scenarios to calculate the cost potential savings when already running resources in Azure and FAQs. And lastly, at the bottom, you can click purchasing options for the product(s).

More details of Azure pricing are available on the pricing landing page.

Considerations Cost Calculator

An Azure calculator is a tool for estimating and not actual costs generated by a client when using the products. It depends on the workload, the number of environments, sizing, and support costs (not just from Microsoft itself, yet also the cost of those managing the product from the client-side). Using the tool can be a good starting point to provide the client a feeling of the cost generation of potential workloads that run on the platform. Furthermore, you can also use the tool to perform an overall calculation by including multiple environments, sizing, and support leveraging Excel. In addition, there is also a TCO calculator through the Azure pricing landing page.

Cost Management

The cost management + billing service and features are available in any subscription in the Azure portal. It will allow you to do administrative tasks around billing, set spending thresholds, and proactively analyze azure cost generation. For example, in the Azure Portal, under Cost Management and Billing, you can find Budgets to create a budget for your costs in your subscription. In the create budget, you can define thresholds on actual and forecasted costs, manage an action group, specify emails (recipients for alerts) and language.

Considerations Cost Management

A key aspect regarding cost control is to set up budgets (mentioned earlier) at the beginning once a subscription before workloads land or resources are provisioned to develop cloud solutions. Furthermore, once consumption of Azure resources starts, you can look at recommendations for cost optimizations and Costs Analysis. For instance, the cost analysis (preview) can show the cost per resource group and services.

It is recommended to separate workloads per subscription as per the subscription decision guide. And one of the benefits is splitting out costs and keeping them under control with budgets. And lastly, Azure Advisor can help identify underutilized or unused resources to be optimized or shut down.

Tagging

Tagging Azure resources is a good practice. A tag is a key-value pair and is helpful to identify your resource. You can order your resource with, for instance, a key environment and value dev (development) and a key identifying the department with value marketing. Moreover, you can add various tags (key/values), up to 50. Each tag name (key) is limited to 512 characters and values to 256 characters. More information on limitations is available on the Microsoft docs.

Tagging Considerations

With tags, you can assign helpful information to any resource within your cloud infrastructure – usually information not included in the name of available in the overview of the resource. Tagging is critical for cost management, operations, and management of resources. More details on how to apply them are available in the decision guide. Furthermore, you can enforce tagging through Azure policies – see the Microsoft documentation on policy definitions for tag compliance.

Reporting

Stakeholders in Azure projects will be interested in cost accumulation for workloads in subscriptions. Therefore, reports of resource consumption in the euro, for example, are required. These reports can be viewed in the Azure Portal under Cost Management and Billing. However, you will need filters in the cost analysis or use the preview functionality to be more specific. Or you can export the data to a storage account and hook it up to PowerBI, or use third-party tooling like CloudCtrl.

And finally, as a developer, you can also leverage the available APIs to get costs and usage data. For example, the Azure Consumption APIs give you programmatic access to cost and usage data for your Azure resources. With the data, you can build reports.

Reports considerations

With costs, reports are essential to realize who the target audience is, what information they are looking for and how to present it. In addition, each active resource consumes the Azure infrastructure inside a data center, leading to cost. And cost should represent value in the end. Hence, reporting is critical for stakeholders in your cloud projects. The analysis of costs is in good hands with the cost analysis capabilities; however, the presentation requirements might differ and sometimes require a custom report by leveraging, for instance, PowerBI or a third-party tool.

Wrap up

In this blog post, we discussed Azure cost and hopefully made it clear that developers should care about cost, and they have tools and services available to make life easier. For example, they can set up cost management infrastructure themselves in their dev/test subscriptions if not already enforced or done by IT. Furthermore, they can make IT and the architect(s) aware of it if it is not in place. In the end, I believe it is a shared responsibility of developers and IT responsible for managing the Azure environments/subscriptions.

Cloud Perspectives

Steef-Jan Wiggers

Tag Archives: FinOps