Azure Storage Data-Plane Auth Modes (AAD Default)

[dkuwcreator]

Implements explicit Azure Storage data-plane authentication modes and migrates to Track 2 SDK's while retaining existing columns.

Integration test logs

Logs

Add passing integration test logs here

Example query results

Old plugin

Shared Keys disabled in Azure

> select
  a.subscription_id,
  count(b.name) as blob_count
from azure_innersource.azure_storage_account a
join azure_innersource.azure_storage_blob b
  on b.resource_group = a.resource_group
 and b.storage_account_name = a.name
group by a.subscription_id;


Error: azure_innersource: -> github.com/Azure/azure-storage-blob-go/azblob.newStorageError, /home/runner/go/pkg/mod/github.com/!azure/azure-storage-blob-go@v0.12.0/azblob/zc_storage_error.go:42
===== RESPONSE ERROR (ServiceCode=KeyBasedAuthenticationNotPermitted) =====
Description=Key based authentication is not permitted on this storage account.
RequestId:f3f1ce06-101e-0021-098e-16e202000000
Time:2025-08-26T13:34:01.3551054Z, Details: 
   Code: KeyBasedAuthenticationNotPermitted
   GET https://stwem9.blob.core.windows.net/blob-container-xxxx-0?comp=list&include=copy%2Cdeleted%2Cmetadata%2Csnapshots%2Cuncommittedblobs%2Ctags&restype=container&timeout=61
   Authorization: REDACTED
   User-Agent: [Azure-Storage/0.12 (go1.24.5; linux)]
   X-Ms-Client-Request-Id: [d41f42d2-7b0a-451c-58d7-aaa4bbd1d571]
   X-Ms-Date: [Tue, 26 Aug 2025 13:34:00 GMT]
   X-Ms-Version: [2019-12-12]
   --------------------------------------------------------------------------------
   RESPONSE Status: 403 Key based authentication is not permitted on this storage account.
   Content-Length: [269]
   Content-Type: [application/xml]
   Date: [Tue, 26 Aug 2025 13:34:00 GMT]
   Server: [Microsoft-HTTPAPI/2.0]
   X-Ms-Error-Code: [KeyBasedAuthenticationNotPermitted]
   X-Ms-Request-Id: [f3f1ce06-101e-0021-098e-16e202000000]


 (SQLSTATE HV000)

+-----------------+------------+
| subscription_id | blob_count |
+-----------------+------------+
+-----------------+------------+
> 

Shared Keys enabled in Azure

> select
  a.subscription_id,
  count(b.name) as blob_count
from azure_innersource.azure_storage_account a
join azure_innersource.azure_storage_blob b
  on b.resource_group = a.resource_group
 and b.storage_account_name = a.name
group by a.subscription_id;
+--------------------------------------+------------+
| subscription_id                      | blob_count |
+--------------------------------------+------------+
| 4c121027-3abb-4202-ab42-31573c2a868d | 6          |
+--------------------------------------+------------+

New plugin

With data_plane_auth_mode = "shared_key" and Shared Keys enabled in Azure

> select
  a.subscription_id,
  count(b.name) as blob_count
from azure_innersource.azure_storage_account a
join azure_innersource.azure_storage_blob b
  on b.resource_group = a.resource_group
 and b.storage_account_name = a.name
group by a.subscription_id;
+--------------------------------------+------------+
| subscription_id                      | blob_count |
+--------------------------------------+------------+
| 4c121027-3abb-4202-ab42-31573c2a868d | 6          |
+--------------------------------------+------------+

With data_plane_auth_mode = "shared_key" and Shared Keys disabled in Azure

> select
  a.subscription_id,
  count(b.name) as blob_count
from azure_innersource.azure_storage_account a
join azure_innersource.azure_storage_blob b
  on b.resource_group = a.resource_group
 and b.storage_account_name = a.name
group by a.subscription_id;

Error: azure_innersource: shared key access disabled on storage account (SQLSTATE HV000)

+-----------------+------------+
| subscription_id | blob_count |
+-----------------+------------+
+-----------------+------------+
> 

With data_plane_auth_mode not set and Shared Keys disabled in Azure

> select
  a.subscription_id,
  count(b.name) as blob_count
from azure_innersource.azure_storage_account a
join azure_innersource.azure_storage_blob b
  on b.resource_group = a.resource_group
 and b.storage_account_name = a.name
group by a.subscription_id;
+--------------------------------------+------------+
| subscription_id                      | blob_count |
+--------------------------------------+------------+
| 4c121027-3abb-4202-ab42-31573c2a868d | 6          |
+--------------------------------------+------------+

[Copilot]

Pull Request Overview

This PR implements explicit Azure Storage data-plane authentication modes and migrates from the legacy Track 1 SDK to the modern Track 2 SDK while maintaining existing table schemas. It introduces Azure AD (OAuth) as the default authentication method with intelligent fallback to shared key authentication when needed.

Key changes include:

• Default to Azure AD authentication for storage data-plane operations with automatic fallback to shared key
• Migration from Track 1 (azure-storage-blob-go) to Track 2 SDK (azblob, azqueue)
• Introduction of configurable data_plane_auth_mode parameter with options: auto, aad, shared_key

Reviewed Changes

Copilot reviewed 7 out of 8 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
go.mod	Updates Azure SDK dependencies to Track 2 versions and removes legacy dependencies
azure/table_azure_storage_queue.go	Migrates queue operations to Track 2 SDK with new authentication modes
azure/table_azure_storage_blob.go	Migrates blob operations to Track 2 SDK with new authentication modes
azure/storage_data_plane.go	New centralized authentication logic for storage data-plane operations
azure/connection_config.go	Adds data_plane_auth_mode configuration parameter
README.md	Documents new storage authentication modes and Track 2 SDK adoption
CHANGELOG.md	Documents enhancements and dependency changes

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[ParthaI]

Hello @dkuwcreator,

I’ve reviewed the code changes in this PR — great work! The overall implementation looks solid.

Based on my understanding, I’ve put together the following data flow diagram to reflect the current design:

---

I do have a couple of suggestions for your consideration:

• While I understand the intention, introducing a new config argument for use across 2–3 tables may not be the most appropriate approach, even if it's optional.
• However, from a user perspective, I would expect the authentication logic to work transparently while querying the table, without requiring explicit configuration.
• With that in mind, I’d like to propose a slight design shift to make the behavior more automatic and user-friendly:

Proposed Behavior

1. Check Storage Account Settings: Detect whether shared key access is enabled on the target storage account.

2. Choose the Best Auth Method:

   • Use shared key if enabled (simpler, no additional setup).
   • Fallback to Azure AD if shared key is disabled.

3. Transparent to the User: No config required — authentication is selected automatically based on the storage account’s settings.

Here’s a revised data flow diagram representing the proposed approach:

---

Please let me know if this suggestion aligns with your goals or if you'd like to explore it further.

Thanks again!

[dkuwcreator]

Hey @ParthaI,

Azure recommends using Azure AD (AAD) for data plane access to storage resources (see: https://learn.microsoft.com/en-us/azure/storage/common/authorize-data-access?tabs=blobs). The same pattern applies to other services such as Key Vault. So, where possible, prefer AAD as the default for all data plane operations, not just for blobs and queues. Adopting this will affect additional tables/entities.

The proposed general auth mode configuration makes sense if you don't want to use one or the other auth method:

• auto (default): Attempt AAD first (or shared key first, we still can decide), then fall back to the other if the first fails.
• aad: Use only AAD; never attempt shared key.
• shared_key: Use only shared key; never attempt AAD.

A major part of this change is also migrating from legacy Track 1 (blob, queue, etc) to Track 2 Azure Go SDK packages.

Regarding shared keys: to obtain them you already need a client capable of checking whether keys are enabled. If keys are disabled, proceed with AAD. If enabled, you must still decide whether to prefer them; I recommend AAD by default, while acknowledging some performance or simplicity advantages of shared keys in certain cases.

Because of this decision flow (establish connection, inspect key availability/state, choose auth method), introducing a dedicated service/component responsible for authentication strategy selection would improve clarity and reuse.

[ParthaI]

Hi @dkuwcreator — thanks for the detailed write-up and for pushing the Track 2 SDK migration; that’s a solid step forward and aligns with Azure guidance to prefer AAD for data-plane access.

Concern with a connection config arg data_plane_auth_mode
My concern is with adding a new connection config arg is that it would only affect a small subset of storage tables (blobs/queues/ADLS) rather than the plugin broadly. We try to keep connection config scoped to universally useful options; otherwise, we risk configuration sprawl.

Alternative: internal, policy-driven selection (no user config)
Because we already fetch storage account properties, we have allowSharedKeyAccess available. We could choose the auth path automatically per account without exposing a new knob:

1. Read allowSharedKeyAccess from existing management-plane data.
2. Attempt AAD when available (default) and required.
3. If allowSharedKeyAccess == true and AAD is unavailable/unauthorized, fall back to Shared Key/SAS.
4. If allowSharedKeyAccess == false, never attempt Shared Key/SAS and return a clear diagnostic if the user supplied one.

Implementation notes:

• AAD-first by default (matches Azure guidance), with fallback to Shared Key only if permitted and practical.
• Treat SAS as key-based for strategy selection. If shared-key access is disabled, we should not attempt SAS and should emit a clear message.
• Keep this logic centralized and reused by storage tables (Blobs/Queues/ADLS Gen2). Services like Key Vault remain AAD-only and bypass this chooser.

Benefits

• Zero new configuration for users; behavior adapts to the account’s security posture.
• Security-forward default (AAD-first) while preserving backward compatibility where Shared Key is still allowed and commonly used.
• Less surface area in connection config; fewer plugin-specific flags to learn.

If the community still wants an explicit override, we can add a single, optional plugin-level switch (auth_preference = "auto" | "aad" | "shared_key") later, but I’d propose shipping the internal auto-selection first.

Please let me know if the above message finds you well.

Thanks!