Open
Conversation
Signed-off-by: Elijah Newren <newren@gmail.com>
In partial clones, `git cherry` fetches necessary blobs on-demand one at a time, which can be very slow. We would like to prefetch all necessary blobs upfront. To do so, we need to be able to first figure out which blobs are needed. `git cherry` does its work in a two-phase approach: first computing header-only IDs (based on file paths and modes), then falling back to full content-based IDs only when header-only IDs collide -- or, more accurately, whenever the oidhash() of the header-only object_ids collide. patch-ids.c handles this by creating an ids->patches hashmap that has all the data we need, but the problem is that any attempt to query the hashmap will invoke the patch_id_neq() function on any colliding objects, which causes the on-demand fetching. Insert a new prefetch_cherry_blobs() function before checking for collisions. Use a temporary replacement on the ids->patches.cmpfn in order to enumerate the blobs that would be needed without yet fetching them, and then fetch them all at once, then restore the old ids->patches.cmpfn. Signed-off-by: Elijah Newren <newren@gmail.com>
In partial clones, `git grep` fetches necessary blobs on-demand one at a time, which can be very slow. In partial clones, add an extra preliminary walk over the tree similar to grep_tree() which collects the blobs of interest, and then prefetches them. Signed-off-by: Elijah Newren <newren@gmail.com>
Author
|
/submit |
|
Submitted as pull.2089.git.1776379694.gitgitgadget@gmail.com To fetch this version into To fetch this version to local tag |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Partial clones provide a trade-off for users: avoid downloading blobs upfront, at the expense of needing to download them later as they run other commands. This tradeoff can sometimes incur a more severe cost than expected, particularly if needed blobs are discovered as they are accessed, resulting in downloading blobs one at a time. Some commands like checkout, diff, and merge do batch prefetches of necessary blobs, since that can dramatically reduce the pain of on-demand loading. Extend this ability to two more commands: cherry and grep.
This series was spurred by a report where
git cherryjobs were each doing hundreds of single-blob fetches, at a cost of 3s each. Batching those downloads should dramatically speed up their jobs. (And I decided to fix up git grep similarly while at it.)I'll also note that
git backfillwith revisions and/or pathspecs could also improve things for these users, but sincebackfillis a manual command users would have to run and requires users to try to figure out which data is needed (a challenge in the case of cherry), it still makes sense to provide smarter behavior for folks who don't choose to manually run backfill.Also, correct a documentation typo I noticed in patch-ids.h (related to code I was using for the git cherry fixes) as a preparatory fixup.