check: does --max-duration really need --repository-only?

[mr-raj12]

Follow-up from #9925 (this thread: #9925 (comment)).

Right now borg check --max-duration requires --repository-only. One of the old reasons was that the archives check needed the repository phase to run first so it had a complete repository index to work from.

That is not the case in borg2 anymore. ArchiveChecker.check builds its own chunk index with build_chunkindex_from_repo(...) in src/borg/archive.py, so it does not depend on the repository check having run.

The one reason that still stands is that the archives check has no max_duration support, so a time-boxed run can only really cover the repository pack check.

So the open question is whether we keep the requirement (just documented) or drop it and add time-boxing to the archives check too. Filing this so it does not get lost.

[ThomasWaldmann]

In a similar way as it remembers which packs it has checked when and what the outcome was, it could also remember this for the check archives. Needs some benchmarking first, whether checking an archive takes long enough so this is worthwhile doing.

[mr-raj12]

I benchmarked this. For normal-sized archives a single archive's check is cheap, and --verify-data wouldn't benefit from per-archive tracking at all as it's currently written.

Setup: borg 2.0.0b23 (current master), local repo on ext4, aes256-ocb + sha256. Archives phase isolated with --archives-only; each number is the min of 3 runs.

Sensitivity to archive data size (repo ~1.4 GiB, 4 archives):

archive	meta (s)	verify-data (s)
1 MiB	3.71	9.98
256 MiB	3.39	9.81
1 GiB	3.62	9.84
20k files	4.04	10.50

Sensitivity to item count (metadata check, tiny files):

files	meta (s)	marginal (s)	ms / 1k files
1	11.63	—	—
10k	11.89	+0.26	26
50k	12.35	+0.72	14
100k	14.02	+2.39	24
200k	16.25	+4.62	23

The baseline differs between the two tables because the fixed cost scales with total repo item count, not with the selected archive.

Three things stand out:

• Most of the archives-check time is a fixed per-invocation cost: building the chunk index from the repo. A per-archive resume doesn't save this, it's paid once per borg check no matter what -a selects.
• --verify-data is effectively repo-global. The time stayed ~10s whether the selected archive was 1 MiB or 1 GiB, same as checking the whole repo. Resume tracking can't help that phase unless it's first reworked to verify per-archive.
• The one thing that scales per archive is item count in the metadata check, roughly linear at ~23 ms per 1000 files. That extrapolates to ~23s at 1M files and ~2min at 5M files.

So the tracking pays off only for archives with millions of items. Below that, the fixed index-build cost swamps the per-archive part, and simple time-boxing of the archives check (with that fixed cost counted against the budget) would cover it.

[ThomasWaldmann]

~23s at 1M files

At 1000 archives, this would be 23.000s = 6h, so we are getting into "it's worth it" territory there.

If there are more files per archive, it happens even earlier than 1000 archives.