Skip to content

feat: deduplicate full-import health validation - #833

Open
nrlcode wants to merge 1 commit into
javi11:mainfrom
nrlcode:feat/full-import-health-dedup
Open

feat: deduplicate full-import health validation#833
nrlcode wants to merge 1 commit into
javi11:mainfrom
nrlcode:feat/full-import-health-dedup

Conversation

@nrlcode

@nrlcode nrlcode commented Aug 22, 2026

Copy link
Copy Markdown

What this PR is for

A successful full import already performs a complete 100% STAT validation, but the importer could immediately schedule another health STAT for the same fully validated file. This PR carries a short-lived, exact-segment validation receipt into post-import health scheduling so that redundant check is skipped only when the evidence still matches.

In practical terms: it removes duplicate provider work after a successful complete import without weakening normal health checking.

What changes

  • Carry the exact segment IDs covered by a successful 100% import STAT validation as a transient receipt.
  • Initialize only fully covered imported files as healthy with their normal future check schedule.
  • Avoid the immediate duplicate health STAT for files covered by that receipt.
  • Preserve the normal health-check path for partial, sampled, degraded, stale, mismatched, re-imported, PAR2, or uncertain coverage.

What this deliberately does not change

  • It does not add persistent availability caching.
  • It does not change provider behavior, streaming, article reads, importer validation semantics, or external configuration.
  • It does not treat degraded/tolerant imports as complete validation.
  • It does not address the unrelated metadata directory-mtime test or existing API lint findings.

Verification

  • Focused importer, scheduler/database, health, package, race, and build checks: PASS
  • git diff --check: PASS
  • make build: PASS
  • make test: blocked by the pre-existing internal/metadata/TestDirectoryModTime_StableAcrossHealthSweep failure, reproduced on the clean base.
  • make lint / make check: blocked by three pre-existing findings in unchanged internal/api files.

The metadata test and API lint/check findings are intentionally excluded from this PR and should be handled separately.

@javi11

javi11 commented Aug 23, 2026

Copy link
Copy Markdown
Owner

Wouldn't be sufficient just don't run a instant health check on import and just wait for the next scheduled check?

@nrlcode

nrlcode commented Aug 23, 2026

Copy link
Copy Markdown
Author

lol yes this is probably much simpler. I can still see the reason to do a health check after import, like if you only check 1% of segments on import then it is probably worth health checking soon after import. But we definitely don't need it if your import percentage check is equal to or higher than your health checks.

@nrlcode

nrlcode commented Aug 23, 2026

Copy link
Copy Markdown
Author

Thinking about this more, leaving import percentage extremely low like 1% and then having the health check run quickly after import might be better?

Then if you really needed to start streaming quickly you could get started, and then repair if it came back unhealthy since it should get caught by the health check soon after.

I'll test it out and see how that's working and might close this out if that is easier and I feel like recommended behavior.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants