VaultRecon: An Azure Control Plane/Data Plane Isolation Flaw

Hero Image generated by ChatGPT

This is a personal blog and all content therein is my personal opinion and not that of my employer.

Introduction

In this post, I’m going to talk about an issue I spotted recently within Microsoft Azure.

Before I do though, let me set the scene a little…

What happened?

I was doing some response work in my day job in follow-up to the excellent blog Escalating privileges to read secrets with Azure Key Vault access policies by Katie Knowles at Datadog.

Please go read the full article - its a great read!

While assessing any exposure to the vulnerability described, I was trying to ascertain how to force another Azure service (I’m deliberately not saying which one from an opsec perspective) which dynamically deploys Azure Key Vaults to deploy them with the access configuration set to Azure RBAC rather than Key Vault Access Policies.

In doing so, I wanted to get a view of the underlying configuration to assess what properties may be able to be amended or set in the ARM template to achieve that.

I really dislike ARM, so I opted to use a tool I’ve used before which seems to be relatively unknown even though it has existed since 2015 (and is still in preview!!).

Enter Azure Resource Exposer Explorer

Azure Resource Explorer is effectively an API browser for the Azure Resource Manager REST API, which I’ll refer to as the management API from here on in. I should add there there isn’t a single API, there are Resource Manager APIs for each resource type (each Provider in Microsoft terminology).

It requires authentication just as the Azure admin portal does and is accessible at https://resources.azure.com/.

Once authenticated, you can submit API requests just by navigating the menu tree in the left hand pane, which at the top level has expandable nodes for providers and subscriptions.

Providers here are Azure Resource Manager providers and there is a provider for each service in Azure - the one I’m interested in here being the Microsoft.KeyVault provider.

Everything that the Azure Resource Explorer website does can also be done by directly calling the management API at https://management.azure.com - in fact the management api Uri for the view you are looking at is always provided so that you know how to return the same response via code.

You can also choose to do more than just query information, if your logged in account has the appropriate permissions you can create or update azure resources in Azure Resource Explorer and via the API directly.

As I walked the config of the resource I was interested in, selecting and expanding subscriptions > resourceGroups > resourceGroupName > providers > Microsoft.KeyVault > vaults > keyVaultName I noticed there was a child node called… secrets - we’ll come back to that shortly…

Now, if you’re familiar with the Key Vaults when viewed from within the Azure admin portal (https://portal.azure.com) you’ll know that as long as you have at least the Reader Azure RBAC role at either the resource level or one of the parent levels (Resource Group, Subscription, Management Group, Tenant Root Group) you can read configuration information about the Key Vault.

What you can’t do with that role assigned is create, view, amend or delete any vault objects: Keys, Secrets or Certificates.

If you were to select either of those options in the blade at the left, you will be met with error messages that make it clear you are not authorised.

If you look at the role definition you might be forgiven for thinking you should be able to since it grants all permissions/actions of type read:

{
  "assignableScopes": [
    "/"
  ],
  "description": "View all resources, but does not allow you to make any changes.",
  "id": "/providers/Microsoft.Authorization/roleDefinitions/acdd72a7-3385-48ef-bd42-f606fba81ae7",
  "name": "acdd72a7-3385-48ef-bd42-f606fba81ae7",
  "permissions": [
    {
      "actions": [
        "*/read"
      ],
      "notActions": [],
      "dataActions": [],
      "notDataActions": []
    }
  ],
  "roleName": "Reader",
  "roleType": "BuiltInRole",
  "type": "Microsoft.Authorization/roleDefinitions"
}

However, note that there are no dataActions defined.

Management Plane vs Data Plane

Microsoft, just as many other Cloud Service Providers do, has a concept of isolation of management actions from data actions/access.

This is conceptualised as having a management plane (sometimes called a control plane) and a data plane. The theory being that access to data is separate and isolated from management of the resources that host data (remember the principal of least privilege).l and also Separation of Duties.

Makes sense then that Reader, which is a built-in general purpose RBAC role for read-only access to resources in Azure would not permit data access, yes?

There are specific ways to grant access to the data in a key vault - in both the access models mentioned earlier.

Sticking with RBAC, here’s the definition of the Key Vault Reader role:

{
  "assignableScopes": [
    "/"
  ],
  "description": "Read metadata of key vaults and its certificates, keys, and secrets. Cannot read sensitive values such as secret contents or key material. Only works for key vaults that use the 'Azure role-based access control' permission model.",
  "id": "/providers/Microsoft.Authorization/roleDefinitions/21090545-7ca7-4776-b22c-e363652d74d2",
  "name": "21090545-7ca7-4776-b22c-e363652d74d2",
  "permissions": [
    {
      "actions": [
        "Microsoft.Authorization/*/read",
        "Microsoft.Insights/alertRules/*",
        "Microsoft.Resources/deployments/*",
        "Microsoft.Resources/subscriptions/resourceGroups/read",
        "Microsoft.Support/*",
        "Microsoft.KeyVault/checkNameAvailability/read",
        "Microsoft.KeyVault/deletedVaults/read",
        "Microsoft.KeyVault/locations/*/read",
        "Microsoft.KeyVault/vaults/*/read",
        "Microsoft.KeyVault/operations/read"
      ],
      "notActions": [],
      "dataActions": [
        "Microsoft.KeyVault/vaults/*/read",
        "Microsoft.KeyVault/vaults/secrets/readMetadata/action"
      ],
      "notDataActions": []
    }
  ],
  "roleName": "Key Vault Reader",
  "roleType": "BuiltInRole",
  "type": "Microsoft.Authorization/roleDefinitions"
}

You’ll note that there are two specific dataActions defined here:

• Microsoft.KeyVault/vaults/*/read
• Microsoft.KeyVault/vaults/secrets/readMetadata/action

The first one allows the read of the actual secret values of all types of object in the key vault.

The second one allows the read of metadata e.g. data that describes the secret data (such as creation date, state etc).

So there is a clear precedent here that data that describes the data is, in fact, still data and should not be seen as being the same as configuration data for the resource - the vault.

If it feels like that was obvious and I didn’t need to explicitly call it out, hold on tight…

The Issue

Remember I mentioned there was a child node under the vault in Azure Resource Explorer called secrets?

Hold that thought and let’s review the Key Vault Access Model

Access to a key vault is controlled through two interfaces: the management plane and the data plane.

The management plane is where you manage Key Vault itself.

The data plane is where you work with the data stored in a key vault. For authorization, the management plane uses Azure role-based access control (Azure RBAC) and the data plane uses a Key Vault access policy and Azure RBAC for Key Vault data plane operations.

The table on this page also makes the clear distinction that the data plane is via the vault.azure.net API endpoint whereas the management plane is via the management.azure.com endpoint.

So then why is the management plane, using Azure RBAC, with a Reader role with NO dataActions defined returning the metadata of every secret in the vault?

Microsoft’s Response

I raised this with Microsoft in two distinct ways:

Microsoft Security Response Center

Timeline

• 19th December 2024 @ 14:22 - Raised a submission to MSRC - VULN-143948: Azure Management API Exposes KeyVault Secrets Metadata in breach of RBAC
• 23rd December 2024 @ 10:34 - Status changed to Review/Repro.
• 29th January 2025 @ 03:41 - Response from MSRC (note that their suggested mitigating custom role is directly lifted from my draft blog post I had already shared with them!):

We have completed our analysis and concluded that this is not vulnerability.

Please find analysis details below:

To provide Azure customers with secure ways to reference their Key Vault object URLs via ARM, Key Vault may expose certain object metadata to customers who have been granted the following control plane permissions:

Microsoft.KeyVault/vaults/keys/read Microsoft.KeyVault/vaults/secrets/read Microsoft.KeyVault/vaults/keys/versions/read

It is important to note that these permissions in the control plane do not expose an object’s value. Customers are required to have data action permissions to retrieve an object’s value.

To prevent the inadvertent exposure of sensitive information in object IDs and metadata, we recommend that customers follow naming best practices as outlined in our public documentation. See Objects, identifiers, and versioning for details.

We do recognize the potential risks of overexposure if a privileged customer account is compromised. Therefore, we will take steps to publicly document this scenario to raise awareness and provide guidance for customers who may not require access to Key Vault object URLs via ARM. Additionally, we committed to investigate long-term solutions to limit the overexposure of metadata for customers who have Reader access.

In the interim, customers who do not need to reference secrets or keys via ARM can create a custom role with the following notActions defined, as shown in the sample role definition below:

{
    "properties": {
    "roleName": "Global Control Plane Reader",
    "description": "",
    "assignableScopes": [
      "/subscriptions/{subscriptionId}"
    ],
    "permissions": [
      {
        "actions": [
          "*/read"
        ],
        "notActions": [
          "Microsoft.KeyVault/vaults/keys/read",
          "Microsoft.KeyVault/vaults/keys/versions/read",
          "Microsoft.KeyVault/vaults/secrets/read"
        ],
        "dataActions": [],
        "notDataActions": []
      }
    ]
    }
}

• 29th January 2025 @ 09:30 - I challenged MSRC:

  “Although you say this isnt a vulnerability, you are also saying you will make changes going forward. This therefore to me says, it is a vulnerability.”

• 8th February 2025 @ 08:51 - Final response from MSRC:

  “We understand that the assessment results may have been disappointing for you. We are truly grateful for the time and effort you invested in reaching out to us and sharing your report. Please continue your research, and we look forward to your future contributions.”

Microsoft Unified Support

Timeline

• 19th December 2024 @ 15:47 - Raised a Severity A Unified Support ticket with Microsoft via my employer.
• 19th December 2024 @ 16:22 - Initial contact from engineer asking clarifying questions in order to be able to reproduce the issue.
• 19th December 2024 @ 21:15 - Microsoft unable to reproduce issue, escalating to Escalation Engineers.
• 20th December 2024 @ 15:49 - Microsoft respond with statement that this is expected behaviour and by design, citing(sic):

“When accessing the Object via the Portal, you are using the Data Pane and will use the Access Configuration set on the vault either Access Policies or RBAC. When using the following API (cites the management API) - This expected behaviour as it is connecting through the Control/Management Plane with ARM. Access controls like Access Policies or Key Vault Firewall will not be evaluated for this type of connection only RBAC will be assessed”*. They then cite this document - Key Vault Access Model Overview. I will expand further on this later. Through continued discussion they state that the management api only returns “Basic data” (still data!) and that the “Sensitive data” is only returned via the data plane which is the KeyVault API at vault.azure.net

• 20th December 2024 @ 20:57 - Despite continued discussion with the support engineer and escalation engineer, they close the case stating there is nothing more they can do as any time this has been raised before and has gone to the Product Group (PG) - every time it comes back as being by design. I did state that it being intentional doesn’t make it a good idea, but that didn’t change anything!

My Thoughts On The Microsoft Response

The fact that Microsoft dismiss as being by design and therefore not a vulnerability, whilst then talking about documenting the scenario described, doesn’t sit well with me.

In addition to my earlier unpicking of the different roles and permissions, let’s look deeper into the Access Model Overview that Microsoft were so sure dismisses my concerns…

Access to a key vault is controlled through two interfaces: the management plane and the data plane. The management plane is where you manage Key Vault itself. Operations in this plane include creating and deleting key vaults, retrieving Key Vault properties, and updating access policies. The data plane is where you work with the data stored in a key vault. You can add, delete, and modify keys, secrets, and certificates.

So - that backs up my attestation that you should see nothing at all about the data from the management plan, not even the metadata about the secrets.

Further down that page is the following table:

Again that makes it very clear that there should be no access to the vault objects from the management plane

When I pushed further on that one, they shared AzureDiagnostics logs showing that the data plane accesses are recorded as one of the following event types:

• KeyList
• SecretList
• CertificateList

Whereas accesses via the control plane are recorded as one of the following event types:

• KeyResourceList
• SecretResourceList
• CertificateResourceList

Again though - this is just confirming data access from the control plane - what Microsoft deemed “Basic data”.

Don’t get me wrong - I’m glad it doesn’t expose the actual secret values!

I do suspect this might be a hangover from the older Access Policies way of doing authorisation and the fact that it’s still possible to use that model rather than RBAC, but that isn’t an excuse for what I see as an insecure design choice.

Why Does This Matter?

1. It’s not widely known/advertised
2. It does not conform to a least-privilege model - managing the resource should never allow data access - not even data that describes the data
3. Exposing this data allows an attacker to enumerate potentially every secret in every vault in the tenant if they have Reader (or an equivalent role with e.g. */Read as a defined permission)
4. One of the most common cloud compromise vectors is credential theft - therefore allowing an attacker (whether malicious insider or a criminal using a stolen credential) to enumerate all your secrets then allows them to decide based on the metadata which secrets in which vaults might be most worth the effort to try other attack paths in order to gain access to the secret
5. YOU CANNOT BLOCK THIS ACTIVITY
6. YOU CANNOT BLOCK THIS ACTIVITY

On that last point (repeated because its important), I played around with a few ways of trying to remediate the risk:

Blocking Access to Azure Resource Explorer

You could block access to Azure Resource Explorer and it’s pretty easy - since I don’t have any enterprise tooling/EDR on my personal machine I have tested just by adding an entry to the hosts file on my Windows machine using the following Powershell one-liner to redirect it to my local machine:

Add-Content -Path C:\Windows\System32\drivers\etc\hosts -Value "127.0.0.1 resources.azure.com"

This will leave the bottom half of your hosts file looking something like this (You need to have local Administrator privileges to edit this file - I done this from an elevated powershell session and also used that to open the file in notepad afterwards to confirm the change - notepad C:\Windows\System32\drivers\etc\hosts):

# localhost name resolution is handled within DNS itself.
#	127.0.0.1       localhost
#	::1             localhost

127.0.0.1 view-localhost # view localhost server
127.0.0.1 resources.azure.com

This does block access to the Azure Resource Explorer:

However, this doesn’t achieve much since, as I mentioned way back at the start, it’s just an API browser front-end for the management.azure.com management API.

So let’s block that then? (If you already know what’s gonna happen, shh…)

Blocking Access to management.azure.com

This time I ran the following powershell one-liner to also redirect management.azure.com to my local machine:

Add-Content -Path C:\Windows\System32\drivers\etc\hosts -Value "127.0.0.1 management.azure.com"

# localhost name resolution is handled within DNS itself.
#	127.0.0.1       localhost
#	::1             localhost

127.0.0.1 view-localhost # view localhost server
127.0.0.1 management.azure.com

Unfortunately - this API is fundamental to Azure - if you block it you will break the portal, you will break ARM deployments, Terraform deployments of azure resources and Az Powershell/Az CLI.

Don’t believe me? Lets reload the Azure portal after adding this hosts file entry:

Not good huh?!

Azure Resource Explorer - End Of Life Announcement

On the 13th January 2025, I noticed something new:

A banner at the top of Azure Resource Explorer stating that “Resource Explorer is shutting down”…

Although the timing seems suspicious, the link takes you to a Github issue raised on June 28th 2024 stating:

Azure Resource Explorer - resources.azure.com will be shutting down soon. We recommend alternatives such as https://github.com/projectkudu/ARMClient or the Azure Portal for managing Azure resources.

The github repo will still be available.

Thank you.

Minimising The Risk

There are things that we can do which will reduce our risk a little.

Attack Surface Reduction - Who Really Needs Reader?

Review your Azure RBAC permissions to make sure that the principals (human and non-human alike) who have the Reader role (or a role that also has */read or similar) are kept to as few as you need in order for your business to operate

Attack Surface Reduction - Tighten The Scope

Review your Azure RBAC permissions to make sure that the scope of the above permissions is as tight as possible - don’t grant those permissions at Management Group, Subscription or even Resource Group level if you don’t need to - allow it only at resource level if at all possible

Attack Surface Reduction - Reduce Standing Privilege

If you have an Microsoft Entra P2 or M365 E5 licence you can leverage Entra Privileged Identity Management to reduce standing privilege by requiring those that do need the access to have to activate a PIM assignment that grants temporary access (time-bound).

You CANNOT use on-premise Active Directory groups synced with Entra

You may be familiar with PIM in terms of Just-In-Time(JIT) access to Entra Roles but it has also had the ability to do the same with Groups for quite some time now (originally known as Privileged Access Groups but now called PIM For Groups).

A really important stipulation here - You CANNOT use on-premise Active Directory groups synced with Entra - the group must either of:

• Microsoft Entra Security Group
• M365 Group

Assigning the role to members of a group managed by PIM (with an Eligible rather than Permanent assignment means that by default nobody is in the group unless and until they successfully activate and elevate into the group.

You can require approval from other people upon activation although I would caution against this for lesser privileged and more frequently used roles or groups as it can lead to approval fatigue.

You can require MFA upon activation but remember that this doesn’t guarantee the user is presented with an MFA challenge on every activation request - if the user has a valid authentication token with a valid MFA claim in it, they won’t be re-MFAed before their token is due to expire.

You can use authenticationContexts instead to require step-up authentication if you need this - e.g. require a different form of authentication method from the users’ default method (require FIDO2 for example even if there is already a valid MFA claim from the Authenticator app).

Another caveat - you would now have to monitor for any abuse of the PIM config to escalate privilege.

Custom Roles

Whilst I believe this a fundamental flaw in Microsoft RBAC in Azure that they themselves should fix, there is a workaround I’ve tested that entirely mitigates this issue.

You could create a new custom role, cloned from the Reader built-in role but crucially with some *notActions added as shown in the role definition below:

{
  "properties": {
    "roleName": "Global Control Plane Reader",
    "description": "",
    "assignableScopes": [
      "/subscriptions/{subscriptionId}"
    ],
    "permissions": [
      {
        "actions": [
          "*/read"
        ],
        "notActions": [
          "Microsoft.KeyVault/vaults/keys/read",
          "Microsoft.KeyVault/vaults/keys/versions/read",
          "Microsoft.KeyVault/vaults/secrets/read"
        ],
        "dataActions": [],
        "notDataActions": [
          "Microsoft.KeyVault/vaults/certificates/read"
        ]
      }
    ]
  }
}

This works on the basis that because of the failure by Microsoft to prevent data access from the control plane, we need to subvert the wildcard permission in the Reader built-in role by explicitly blocking permissions that grant data access in the data plane.

Whilst this does work, it shifts the pain from Microsoft as the Cloud Service Provider to you as the customer.

You’d be letting them off the hook for failing to uphold their side of the shared responsibility model whilst the burden of working out what permissions to block, and keeping that up to date as the cloud services evolve has now fallen on you as you maintain custom roles, not Microsoft.

Detection of Enumeration

Implement detection to help you detect potentially malicious enumeration of secrets through this route.

To that last point - below is a KQL query you can use to detect this activity if you have your keyvault diagnostic logs configured for all your key vaults and being ingested into a Log Analytics Workspace:

AzureDiagnostics
| where ResourceType == "VAULTS" and ResourceId contains "Microsoft.KeyVault/vaults"
| where TimeGenerated > ago(365d)
| where requestUri_s contains "management.azure.com"
| where clientInfo_s != "azure-resource-manager/2.0"
| where OperationName in ("KeyResourceList", "SecretResourceList", "CertificateResourceList")
| where httpStatusCode_d == 200

What this does is extract from the AzureDiagnostics table only events where:

• The events are diagnostic logs from the Microsoft.KeyVaults/vaults provider
• The call originated from the control plane e.g. management.azure.com - lets ignore the data plane e.g. .vault.azure.net
• Exclude calls from ARM (I know, perhaps someone can spoof the client info but we have to start somewhere!)
• Only return events that correspond with secrets enumeration from the management plane (*ResourceList)
• Requests that were successful (if you remove this and the other conditions, you will see HTTP 403 where access was denied in the Azure Portal)

You can then use this to configure an Alert Rule in Sentinel that would alert on these - which should be fairly low volume so false positives should be low.

Sample alert rule definition:

{
    "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "workspace": {
            "type": "String"
        }
    },
    "resources": [
        {
            "id": "[concat(resourceId('Microsoft.OperationalInsights/workspaces/providers', parameters('workspace'), 'Microsoft.SecurityInsights'),'/alertRules/a4d34cd4-1564-400e-803a-730bdbb97dc0')]",
            "name": "[concat(parameters('workspace'),'/Microsoft.SecurityInsights/a4d34cd4-1564-400e-803a-730bdbb97dc0')]",
            "type": "Microsoft.OperationalInsights/workspaces/providers/alertRules",
            "kind": "Scheduled",
            "apiVersion": "2023-12-01-preview",
            "properties": {
                "displayName": "Azure_KeyVault_Secret_Metadata_Enumeration_From_Management_Plane",
                "description": "Enumeration of metadata in Azure Key Vaults via Management API which can lead credential theft through allowing an attacker to target the most interesting secrets in a tenant.",
                "severity": "Medium",
                "enabled": true,
                "query": "AzureDiagnostics\n| where ResourceType == \"VAULTS\" and ResourceId contains \"Microsoft.KeyVault/vaults\"\n| where TimeGenerated > ago(365d)\n| where requestUri_s contains \"management.azure.com\"\n| where clientInfo_s != \"azure-resource-manager/2.0\"\n| where OperationName in (\"KeyResourceList\", \"SecretResourceList\", \"CertificateResourceList\")\n| where httpStatusCode_d == 200",
                "queryFrequency": "PT5M",
                "queryPeriod": "PT3H",
                "triggerOperator": "GreaterThan",
                "triggerThreshold": 5,
                "suppressionDuration": "PT5H",
                "suppressionEnabled": false,
                "startTimeUtc": null,
                "tactics": [
                    "ResourceDevelopment",
                    "Reconnaissance",
                    "Execution",
                    "Persistence",
                    "PrivilegeEscalation",
                    "DefenseEvasion",
                    "CredentialAccess",
                    "Discovery",
                    "Exfiltration",
                    "Impact",
                    "LateralMovement",
                    "Evasion"
                ],
                "techniques": [
                    "T1586",
                    "T1591",
                    "T1595",
                    "T0834",
                    "T0871",
                    "T1106",
                    "T1575",
                    "T1651",
                    "T0859",
                    "T1078",
                    "T1404",
                    "T1552",
                    "T1555",
                    "T1649",
                    "T1087",
                    "T1567",
                    "T0827",
                    "T0828",
                    "T0831",
                    "T0832",
                    "T0826",
                    "T0815",
                    "T0829",
                    "T0882",
                    "T0812",
                    "T0849"
                ],
                "subTechniques": [
                    "T1552.005",
                    "T1552.004",
                    "T1555.005"
                ],
                "alertRuleTemplateName": null,
                "incidentConfiguration": {
                    "createIncident": true,
                    "groupingConfiguration": {
                        "enabled": true,
                        "reopenClosedIncident": false,
                        "lookbackDuration": "PT5H",
                        "matchingMethod": "AllEntities",
                        "groupByEntities": [],
                        "groupByAlertDetails": [],
                        "groupByCustomDetails": []
                    }
                },
                "eventGroupingSettings": {
                    "aggregationKind": "AlertPerResult"
                },
                "alertDetailsOverride": null,
                "customDetails": null,
                "entityMappings": null,
                "sentinelEntitiesMappings": null,
                "templateVersion": null
            }
        }
    ]
}

You could then link this to an automation rule to run a logic app, that calls Microsoft Graph and/or Azure APIs to determine, did the user already have access to the secret in the data plane (in which case we might not care) - if they did not have access via the data plane then this is absolutely suspicious and requires investigation/response.

What still concerns me even with the above in place is that you can access the management API from anywhere - though Conditional Access could be used to block external access. The insider risk/compromised account risk persists though.

In addition, who can say that this only exists as an issue for the Microsoft.KeyVault provider?

Is this a wider issue? Stay tuned for my next blog post to find out…

One Last Thing - Exploitation

Now that I’ve walked you through this issue that I’ve named VaultRecon, I have something else to share…

I’ve created an exploit script that will authenticate to your tenant (you should authenticate as a user with Reader as their only Entra or Azure role) and walk through every key vault in every subscription in your tenant and dump out the metadata of every secret in every vault. This then lets you know what is exposed and what might look appealing to an attacker in terms of trying to find another attack path to get at those secrets.

I call it Az-Skywalker and it is available here: https://github.com/Az-Skywalker/Az-Skywalker

Documentation and sample outputs are are covered in the README and a video demonstration is available on my YouTube channel here:

Acknowledgements & Prior Work

Summary - All You Need Is Read(er)

The fact that these issues arise from the Control Plane or Management Plane of the cloud provider APIs and represent (in my personal opinion) failures in RBAC enforcement and control plane/data plane separation adds a deeper dimension to their impact. These are not just generic security flaws; they reflect systemic architectural weaknesses in Microsoft’s cloud infrastructure.

And remember, All You Need Is Read

As ever, thanks for reading and feel free to leave comments down below!

If you like what I do and appreciate the time and effort and expense that goes into my content you can always