feat(filesystem): read Azure through pyarrow.fs - #297
Merged
Conversation
Every glob load pays a per-file open, and that is where the Azure source spent most of its time. Reading `az://`, `adls://` and `abfss://` through `pyarrow.fs.AzureFileSystem` instead of adlfs cuts it: measured against Azurite, opening and reading 300 small blobs costs 2.4 ms/blob through Arrow against 6.7 ms/blob through adlfs, and a 64 MB read runs at 456 MB/s against 193 MB/s. Listing already benefits, because the vendored lister issues one native recursive request for any Arrow-backed client. The native client does not speak the fsspec contract the lister needs, so it goes behind the `_filesystem()` seam S3 introduced, wrapped by an `_AzureArrowFSWrapper` whose `_strip_protocol` removes only the scheme prefix: a blob name may contain `?` or `#` literally, and the generic wrapper reads the rest of the path as a URL. Credentials map field by field. Arrow takes no connection string, so one parse resolves the account, both endpoints and the credential: the account key comes off the parsed client's credential and the SAS off its composed URL, through the same public SDK APIs that already resolve account and endpoint identity. A SAS is normalised to the leading-`?` form, which Arrow requires because it appends the token to the account URL verbatim; either form is accepted on input. `account_host` becomes Arrow's authority-and-scheme pair for both the Blob and the Data Lake endpoint, since Arrow reaches a hierarchical-namespace account through the latter and naming only the former would send half the requests to the public cloud. A suffix carrying a service label yields its sibling by substitution, so a Private Link host pairs correctly, and a connection string that names a `DfsEndpoint` has it used as given rather than derived. An endpoint has to name its account, which is what makes both of Arrow's authority forms derivable, and the endpoint rules S3 applies (HTTP schemes only, no credentials, no query or fragment) carry over. Two parameters lose meaning, and both are refused rather than ignored. `api_version` has no Arrow equivalent, because Arrow pins the version its bundled SDK speaks; an emulator that enforces the version check now needs `--skipApiVersionCheck`, which is how the Azurite test container is started. A source connection string that identifies its account by endpoint alone, with no `AccountName`, cannot be addressed at all. Incremental behaviour and cursor identity are untouched, so nothing resets. adlfs stays for the remote-database staging download and for writes, which dlt's filesystem destination builds itself.
Contributor
|
Important Review skippedAuto reviews are limited based on label configuration. 🏷️ Required labels (at least one) (1)
Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #297 +/- ##
==========================================
+ Coverage 61.66% 61.82% +0.15%
==========================================
Files 235 235
Lines 11408 11487 +79
==========================================
+ Hits 7035 7102 +67
- Misses 4373 4385 +12 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Member
|
Excellent, thank you! Released with v0.11.0. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
az://,adls://,abfss://) throughpyarrow.fs.AzureFileSysteminstead of adlfs, for the per-file open cost every glob load pays.DfsEndpointin a connection string is used as given; otherwise it is derived from the Blob endpoint.Part of #233. S3 and R2 moved in #290; GCS stays on gcsfs, as agreed in that issue.
Behavior changes
Both are refused rather than accepted and ignored, and both are documented in
docs/supported-sources/azure-storage.mdand the changelog.api_versionis no longer accepted on a source URI. Arrow pins the API version its bundled SDK speaks and offers no override. This is not cosmetic: an endpoint that enforces the version check is now unreadable through the source, and Azurite is such an endpoint, so the emulator container is started with--skipApiVersionCheck.SharedAccessSignaturewith a customBlobEndpointand noAccountName) is rejected with a named error.Building a source in Python rather than through a URI, connection arguments now have to be
pyarrow.fs.AzureFileSystemkeywords; an adlfs-only argument is rejected by name rather than ignored.Review
Two independent pre-push reviews completed, both repo-aware, and both findings were applied.
?Arrow requires. Arrow appends the token to the account URL verbatim, and both carriers hand it over without one, so every SAS-authenticated read failed withOutOfRangeInput. Construction cannot catch this, becauseAzureFileSystemperforms no I/O until first use; an emulator test that generates an account SAS and reads through both carriers now does.acct.privatelink.blob.core.windows.net) derived the Blob authority for both services, sending Data Lake requests to the wrong host. The service label is now substituted wherever it sits in the suffix.DfsEndpointwas silently dropped, and the connection string was parsed twice per source build. Both fixed.Changes
pyarrow.fs, behind the_filesystem()seam S3 already uses, with an Azure wrapper whose_strip_protocolremoves only the scheme prefix so a blob name keeps?,#and percent escapes.--skipApiVersionCheck, and drop the adlfs API-version test fixture it replaces.10 files changed, 997 insertions, 100 deletions.
Test plan
az://,adls://andabfss://each read through the same client; blob names carrying#,?and%2Fsurvive discovery and open.pyarrow.fs.AzureFileSystemitself, so a wrong keyword name fails rather than passing.ty check,validate-pyproject._azurefsconfirmed present in the pyarrow 25.0.1 wheels for the CI matrix (cp310 and cp314; manylinux, macOS arm64, win_amd64).Live service-principal authentication is not exercised, here or before this change: the emulator validates only a token's shape, needs HTTPS, and
ClientSecretCredentialwould still have to reach Entra ID for a token.No visual surface changed, so screenshots are not applicable.