Key Vault Managed Identity: The Secret You Should Not Store

Posted on November 10, 2026 by steefjan1970

Every .NET application needs secrets: an API key for a third-party service, a connection string, a signing key. The rule is old and still right. Never hardcode them, and never commit them. On Azure, Key Vault managed identity is the standard answer, and the usual summary fits in four steps: store the secret, enable a managed identity, grant it access, read the secret from code.

Those four steps are correct. However, they are also where most articles stop, and where the interesting decisions start. In this post, I build the pattern end to end on Azure. Then I measure the part the four steps skip: how a rotated secret actually reaches a running application, and what each option costs. Finally, I argue that the most common example, the database password, is the one secret that should not be in the vault at all.

The companion repo deploys everything with azd up and includes the measurement scripts.

The Key Vault managed identity reference architecture

The sample is a small ASP.NET Core 8 API on App Service (Linux). It has a user-assigned managed identity called id-api. Key Vault runs in RBAC mode and holds one secret, ExternalApiKey, the key for a fictional third-party API. The identity has the Key Vault Secrets User role on the vault and nothing else. Azure SQL (serverless) accepts Entra authentication only. In addition, every read of the vault is written to Log Analytics as an AuditEvent.

Three choices that matter

Three choices in that picture deserve a sentence each.

User-assigned, not system-assigned. A system-assigned identity doesn’t exist until the web app does. As a result, “grant access before deploy” is impossible, and every redeploy becomes a race. A user-assigned identity, on the other hand, has a known client id before the app is created, so Bicep can assign roles and set keyVaultReferenceIdentity in one pass. The app settings carry AZURE_CLIENT_ID, which tells DefaultAzureCredential which identity to use. If the token flow behind that is new to you, I took it apart in [INTERNAL LINK 1: Microsoft Entra ID under the hood].

RBAC, not access policies. Role assignments are Azure resources, so they live in Bicep and show up in Activity Log. “Secrets User” means read only. Meanwhile, the person who runs azd up gets “Secrets Officer”, so rotation and local development work with the same vault.

Audit on from day one. Key Vault’s diagnostic setting is the cheapest security control you can turn on. It answers “which identity read which secret, and when” with one query. The repo ships the queries in measure/audit.kql.

Four ways a secret reaches your code

Here is the part the four-step summary hides. “Configure your application to retrieve secrets” is not one thing. In a single App Service instance, there are four distinct paths, and they behave differently under rotation.

1. Key Vault reference in app settings. The app setting holds @Microsoft.KeyVault(VaultName=kv;SecretName=ExternalApiKey). App Service resolves it with keyVaultReferenceIdentity before your process starts, so the code sees a plain environment variable. No SDK, and no vault call on the request path. The price: App Service caches the value and re-fetches it every 24 hours, or on a configuration change. What a restart, a deployment, and the refresh API do to a running app is less clear from the documentation, so I measured it below.

2. The configuration provider. AddAzureKeyVault from Azure.Extensions.AspNetCore.Configuration.Secrets loads secrets into IConfiguration at startup and re-reads them on ReloadInterval. Note that it loads every secret the identity may read in that vault. In other words, the scope of the role, not the code, decides what ends up in memory. Startup fails if the vault is unreachable.

3. SecretClient behind a cache. GetSecretAsync with an IMemoryCache and a TTL. One vault call per TTL window per instance, whatever the request rate. At least, that is the theory; the measurement below has a surprise.

4. SecretClient on every request. Always current. Also one SecretGet per request, against a limit of 4,000 vault transactions per 10 seconds per vault. A busy API hits that ceiling, and when it does, Key Vault answers 429 for every caller of that vault, including the other apps that share it.

One endpoint, four fingerprints

The sample exposes all four behind one endpoint, GET /api/secrets/status. It returns a fingerprint (a truncated SHA-256) of the value each path currently holds, never the value itself. That makes rotation measurable.

Measuring rotation with Key Vault managed identity

measure/rotate.py writes a new version of ExternalApiKey with az keyvault secret set. Then it polls the status endpoint every 10 seconds and records when each path first serves the new value. I ran it four times, each time with a different action on the reference path afterwards.

Path	Seen after rotation (three clean runs)	What decides it
4 on demand	0 s, 0 s, 0 s	nothing, every request reads the vault
3 cached (60 s TTL)	60 s, 65 s, 44 s	where the rotation falls inside the TTL window
2 provider (5 min reload)	44 s, 33 s, 119 s	where the rotation falls inside the reload window
1 reference	not within 15 min	see below

The on demand path is immediate. The cached and provider paths, however, land anywhere inside their window. The 5 minute reload interval is an upper bound; three rotations happened to fall 33 to 119 seconds before the timer fired. Therefore, treat the interval as the worst case when you size the overlap between an old and a new secret.

The reference path: four actions, one result each

The reference path is where the documentation and the measurement part ways. I tried four actions:

The refresh API (POST .../config/configreferences/appsettings/refresh) reported Resolved. Still, the process kept serving the old value for the full 10-minute window.
az webapp restart: old value, 10 minutes.
A new deployment (azd deploy api): old value.
A configuration change (one new app setting): new value after 99 seconds, including the restart the change triggers.

So on this app, in this region, on this day, the only way to get a rotated secret into a Key Vault reference within the day was to change the app’s configuration. I didn’t wait for the documented 24-hour re-fetch. Your results may differ, and that is exactly the argument for the script: run it against your own app before you rely on a reference for a rotating secret.

Rotation is a property of the reading path

The point is that “rotation” belongs to the reading path, not to Key Vault. You can rotate a secret in the vault in one second. Whether the application notices in one second, two minutes, or tomorrow depends on a decision that was made in Program.cs or in an app setting, usually without anyone writing it down.

So write it down. For each secret, state the rotation window the application tolerates, and pick the path that meets it. In practice that means:

A reference for secrets that change with a deployment anyway, where the pipeline changes an app setting as part of the rollout.
The provider or a cached client for secrets that rotate on a schedule, with a reload interval shorter than the overlap window of the old and new value.
A per-request read almost never. If you think you need it, you need a cache with a short TTL instead.

Measuring the vault calls

measure/burst.py sends 200 requests, ten at a time, to one path and records the window. The application reports whether a request hit the vault, and the Key Vault audit log in Log Analytics confirms it from the other side. I ran the set twice, twenty minutes apart; both runs gave the same counts.

Path	Requests	Vault calls (app)	Vault calls (audit log)
4 on demand	200	200	200, 0 throttled
3 cached (60 s TTL)	200	10	10, 0 throttled
2 provider	200	0	no rows in the window

Ten calls, not one

Two things in that table are worth more than the headline.

First, the cached path made ten calls, not one. The cache was empty when the burst started, ten requests arrived at once, all ten missed, and consequently all ten fetched. IMemoryCache.GetOrCreateAsync does not serialize concurrent misses. That is a cache stampede in miniature, and it repeats after every restart and every TTL expiry. With 50 instances and a 60 second TTL, for example, it is 50 calls a minute at rest and a burst of several hundred the moment a deployment rolls. The fix is a lock around the miss, with SemaphoreSlim or with .NET 9’s HybridCache, which does this for you.

Second, the audit log shows the client id of the managed identity on every row, with the result code. That is the row you want when a security reviewer asks who read production secrets this month. Likewise, it is where a 429 shows up first when a per-request path starts to push against the 4,000 transactions per 10 seconds limit.

End-to-end latency from my client was 730 to 770 ms at p50 for all three paths, which says more about the distance to Sweden Central than about Key Vault. The number that matters is on the server: a vault read from App Service in the same region took 107 ms warm and 194 ms cold, and that cost sits on the request path for path 4 only.

The secret you should not store

Now the example that every introduction to Key Vault uses: the database password. The API connects to Azure SQL, so put the password in Key Vault and read it with the managed identity.

Look at that sentence again. The application already has an identity that Entra trusts enough to hand out tokens for Key Vault. Azure SQL trusts the same Entra. So the application can log in to the database as that identity, with a token, and there is no password to store, rotate, leak, or audit.

In the sample, the connection string is plain text in app settings, because nothing in it is secret:

			
Server=tcp:<server>.database.windows.net,1433;Database=orders;Authentication=Active Directory Default;Encrypt=True;

Microsoft.Data.SqlClient asks Entra for a token for id-api and presents it to SQL. On the database side, a contained user exists for the identity, created from its client id:

			
CREATE USER [id-api] WITH SID = 0x..., TYPE = E;
ALTER ROLE db_datareader ADD MEMBER [id-api];

The repo creates that user in a post-provision step, as the SQL Entra admin, with an access token from az. There is no SQL password anywhere: not in the repository, not in the deployment, and not in the vault. Moreover, the SQL server has azureADOnlyAuthentication set to true, so a password login cannot even be created by accident.

GET /api/orders proves it. The response shows SUSER_SNAME() as the identity (<client id>@<tenant id>, the form SQL uses for a user created by SID), three rows, and a 9 ms connection open. Nothing was read from Key Vault to get there.

What “wherever possible” actually means

This is what “use managed identity wherever possible” means in practice. It does not mean “use managed identity to reach Key Vault”. It means that for every Azure service that accepts Entra tokens (SQL, Storage, Service Bus, Event Hubs, Cosmos DB, Azure OpenAI, App Configuration), the secret should not exist. Key Vault is for the remaining cases: keys for third-party APIs, certificates, and the odd legacy system that only speaks passwords. In short, Key Vault managed identity is the pattern for the secrets you cannot avoid, not a reason to keep the ones you can.

Where Key Vault managed identity is the wrong answer

Secrets that change per request or per tenant. Key Vault is a configuration store, not a runtime lookup. The 4,000 transactions per 10 seconds limit is per vault, shared by every app that uses it.
Local development through the vault. DefaultAzureCredential makes it work, but the audit log will then show every developer reading production secrets. Use a separate vault per environment, or user secrets locally.
Key Vault references for Application Insights settings. The portal reads those settings directly and cannot resolve references, so the telemetry views go blank.
A vault with the firewall set to the app’s outbound IPs. The documented pattern is a private endpoint and vnetRouteAllEnabled, because outbound IPs of App Service change. I covered the network side of that in [INTERNAL LINK 2: Zero Trust on Azure].
One vault for the whole company. RBAC on secrets is per vault, and the provider path loads everything the identity may read. Small vaults, one per application and environment, keep both the blast radius and the reload payload small.

What I would check first

If you inherit an application that “uses Key Vault”, these four questions find most of the problems within an hour:

Which of the four paths does it use, and what rotation window does that imply? Rotate a test secret and watch.
Does the vault hold a password for an Azure service that accepts Entra tokens? If so, replace it with the identity.
Is AuditEvent flowing to a workspace, and has anyone ever queried it?
Is the vault firewall or the role scope wider than the one application that needs it?

The sample repo answers all four for its own setup, and azd down --purge removes it in a few minutes. It follows the same build, deploy, and measure pattern as the previous blog post on Building Scalable Solutions on Azure, so the two repos sit side by side.

Sources

Secret Management in the Cloud

Posted on January 22, 2022 by steefjan1970

I have been using Azure Key Vault for secret management for the last two or three years in my projects or advice my peers, client, and colleagues I work with to do so. Azure Key Vault is a service that provides storing and managing secrets with policies and the ability to access them using .NET code. Moreover, it is not just .NET yet also a service principal that can access it to get a secret for establishing a connection or a pipeline. The secrets can be API keys, connection strings, credentials, certificates, etc. I like to discuss a secret management use case in this blog post and dive into its details.

Use case Key Vault and D365 FO Business Events

In a recent project regarding unlocking data from a Dynamics 365 Finance and Operations (FO) instance, I leveraged the concept of Business Events, where a Logic App subscribes to a specific event published on a custom Event Grid Topic. Let me further explain the scenario and where Key Vault comes into play. Below you see a diagram of integration between D365 FO and third party system. The latter receives data from D365 based upon a specific business event.

Within D365 FO, you can define a destination for a business event. As shown in the diagram, the destination is an Event Grid Topic. When following the Microsoft documentation of Business Events and Event Grid, you will notice that a Key Vault is required to keep the access key of the Event Grid Topic as a secret. Furthermore, you will need to create a so-called App registration in

Azure Active Directory. Azure App registrations are a simple and effective way to configure authentication and authorization workflows for many client types. In this case, a client identifying D365 – allowing access to the Key Vault instance to extract the access key for the custom Event Grid Topic.

Once the app registration is in place, the next step is to add it to the access policies in the Key Vault instance. The registration represents D365, and it needs access to the Key Vault to extract the access key for the Azure Event Grid topic. The app registration only requires the Get and List secret permissions to retrieve the Key Vault secrets.

The endpoint configuration is the next step when the app registration and policy are in place, the custom Event Grid topic is available, and its access key is a secret in Key Vault. The screenshot below shows the configuration of an actual endpoint (destination) for the events – the custom Event Grid topic.

For configuring the endpoint (destination), you need to provide a name. So first, the endpoint type is filled in by default, followed by the endpoint URL (destination endpoint – Event Grid topic URL) and then the details for the Key Vault. These details are the client id of the app registration, its secret, the DNS name of the Key Vault instance, and key vault secret name – which has the secret, i.e., access key to the custom Event Grid topic. And finally, you can press Ok for the creation of the endpoint. You can subsequently attach the endpoint to the necessary business event and activate it when the endpoint is created.

Once the endpoint is active and a specific business event is attached to the endpoint, the event will end up with the subscriber – Logic App. An example of a business event is shown below:

{
  “BusinessEventId”: “PurchaseOrderConfirmedBusinessEvent”,
  “ControlNumber”: 5637365024,
  “EventId”: “9D42A382-12E8-48F6-9BB2-29A1G4E39773”,
  “EventTime”: “/Date(1642759229000)/”,
  “LegalEntity”: “fnl1”,
  “MajorVersion”: 0,
  “MinorVersion”: 0,
  “PurchaseJournal”: “PO1-002342-11”,
  “PurchaseOrderDate”: “/Date(1642723200000)/”,
  “PurchaseOrderNumber”: “PO1-002342”,
  “PurchaseType”: “Purch”,
  “TransactionCurrencyAmount”: 1553.46,
  “TransactionCurrencyCode”: “EUR”,
  “VendorAccount”: “IFF1095”
}

The Logic App can use the details to retrieve more information (through OData calls) about the purchase order in this case. And as shown in the diagram, send the enriched json to a service bus queue to handover the another Logic App to transform it into an XML to be sent to an application Basware (provider of software for financial processes, purchase to pay, and financial management).

Managing Key Vault

To properly set up the process around Key Vault and secrets, the administrator (Azure Ops) is responsible for creating the app registration. The administrator will make the app registration and manage the Key Vault. Moreover, the person is also the one in my view that does the endpoint configuration. Therefore, the integration developer will only need to connect the Logic App to the Event Grid topic. Similarly, the SFTP connection requiring credentials or certificates can also leverage the Key Vault and require the same administrator.

The diagram below shows what the administrator can do regarding the app registration and managing the Key Vault instance. Also, the authentication process is shown from the application side – in our case, creating the endpoint from D365. Finally, D365 will use the app registration to authenticate against Azure AD to retrieve a token necessary to access the key vault secret.

I like to point here regarding this scenario that business events might need to be set up again when a database refresh is done. Note that when the endpoint configuration fails, you can see an error like:

Unable to get secret from Key Vault DNS: <dns of the key vault instance> Secret name: <name secret>

In that case, either the app registration client id or secret is wrong, or worse, the app registration is expired (the error messages will not tell you that!). An app registration expires (the max is two years). Hence, be aware that the events when the app registration is expired will not reach the Event Grid topic, and errors will occur on the D365 side. Therefore, I recommend monitoring the expiration for the app registration, and also, the secrets can have an expiry date – so keep an eye on that too!

Other Cloud Public Cloud Providers

Interestingly, Azure is not the only public cloud platform with a secret certificate and key management service. For example, AWS actually has three services – AWS Secrets Management, AWS Certificate Manager, and AWS CloudHSM. With AWS Secrets Manager, users can manage access to secrets using a fine-grained set of policies, control the lifecycle of secrets, and secure and audit secrets centrally. Furthermore, this is a managed service with a pay-as-you-go model available in most AWS regions. Sound familiar? Azure Key Vault is similar, right? Almost, yet Key Vault has most of the capabilities found in the three earlier mentioned AWS Services.

What about the Google Cloud Platform? Well, on GCP, you will find Secret Manager, which also enables users to store and manage secrets, including policies and rotation. Furthermore, the service offers management of certificates. And lastly, the public cloud has a separate service for key management with Key Management Service (KMS).

Cloud Perspectives

Steef-Jan Wiggers

Tag Archives: Azure Key Vault