Skip to content

Makes prefetcher none cache default - #992

Merged
ankitaluthra1 merged 10 commits into
fsspec:mainfrom
ankitaluthra1:make-prefetcher-default
Aug 5, 2026
Merged

Makes prefetcher none cache default#992
ankitaluthra1 merged 10 commits into
fsspec:mainfrom
ankitaluthra1:make-prefetcher-default

Conversation

@ankitaluthra1

Copy link
Copy Markdown
Collaborator

No description provided.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request enables the adaptive concurrent prefetcher by default, updating the default concurrency from 1 to 4 and dynamically setting the default cache type to "none" when prefetching is active (or "readahead" when disabled). The documentation and test suites have been updated to reflect these new defaults. Feedback on the changes suggests handling potential string values for the prefetching kwarg to avoid incorrect boolean evaluation, updating ZonalFile to align with the new dynamic cache defaults, and wrapping the environment variable integer parsing in a try-except block to prevent import-time crashes.

Comment thread gcsfs/core.py Outdated
Comment thread gcsfs/core.py Outdated
Comment thread gcsfs/zb_hns_utils.py Outdated
@ankitaluthra1

Copy link
Copy Markdown
Collaborator Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request enables adaptive concurrent prefetching by default, changing the default concurrency from 1 to 4 and setting the default cache type to "none" when prefetching is active. It also updates the documentation and test suite to reflect these new defaults. Regarding the feedback, a discrepancy was identified in how use_experimental_adaptive_prefetching is resolved in GCSFile.__init__ compared to ZonalFile.__init__. Specifically, passing None explicitly in kwargs would evaluate to False and disable prefetching while keeping cache_type="none", resulting in no caching or prefetching at all. A code suggestion has been provided to resolve this consistently.

Comment thread gcsfs/core.py Outdated
@codecov

codecov Bot commented Jul 31, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 90.47619% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.03%. Comparing base (370ca0b) to head (3d75082).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
gcsfs/zb_hns_utils.py 50.00% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #992      +/-   ##
==========================================
+ Coverage   89.68%   90.03%   +0.34%     
==========================================
  Files          16       16              
  Lines        3579     3593      +14     
==========================================
+ Hits         3210     3235      +25     
+ Misses        369      358      -11     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ankitaluthra1

Copy link
Copy Markdown
Collaborator Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request enables the adaptive concurrent prefetcher by default with a default concurrency of 4 and a default cache type of "none". It refactors GCSFile and ZonalFile to dynamically resolve the cache type and prefetcher settings when not explicitly configured, and updates the documentation and test suite accordingly. Feedback on the changes highlights a potential RuntimeError or deadlock in gcsfs/prefetcher.py when scheduling the asynchronous close task on a running loop that is not the prefetcher's own background loop; a fix is suggested to ensure tasks are scheduled on the correct event loop.

Comment thread gcsfs/prefetcher.py Outdated
@ankitaluthra1

Copy link
Copy Markdown
Collaborator Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request enables adaptive concurrent prefetching by default for GCSFile and ZonalFile by resolving the default cache_type to None (which dynamically defaults to "none" with prefetching active). It also increases the default concurrency from 1 to 4. The review feedback correctly identifies two important issues: first, the accidental removal of the critical GoogleCredentials.load_tokens() call in gcsfs/core.py, and second, the need to validate that DEFAULT_GCSFS_CONCURRENCY is a positive integer to avoid invalid concurrency limits.

Comment thread gcsfs/core.py
Comment thread gcsfs/zb_hns_utils.py
@ankitaluthra1
ankitaluthra1 marked this pull request as ready for review August 5, 2026 14:25
Comment thread docs/source/prefetcher.rst Outdated
Comment thread gcsfs/core.py Outdated
Comment thread gcsfs/core.py
Comment thread gcsfs/core.py Outdated
@ankitaluthra1
ankitaluthra1 merged commit 8880f63 into fsspec:main Aug 5, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants