Skip to content

Generate seed file presigned URL in crawl operator, update prior to expiry - #3600

Open
tw4l wants to merge 11 commits into
mainfrom
issue-3576-seed-file-presigned-url
Open

Generate seed file presigned URL in crawl operator, update prior to expiry#3600
tw4l wants to merge 11 commits into
mainfrom
issue-3576-seed-file-presigned-url

Conversation

@tw4l

@tw4l tw4l commented Aug 25, 2026

Copy link
Copy Markdown
Member

Fixes #3576

Changes

  • Pass seed file id, not presigned url, to crawl job template, and generate presigned URL for seed files in crawl operator as needed
  • Track presigned URL expiry and regenerate as needed before the URL expires

@tw4l
tw4l requested review from ikreymer and mistydemeo August 25, 2026 21:29
@tw4l tw4l changed the title Re-generate seed file presigned URL in crawl operator prior to expiry Generate seed file presigned URL in crawl operator, update prior to expiry Aug 25, 2026

@mistydemeo mistydemeo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Making sense to me. Since this has changed a model, I assume we need a corresponding migration?

@tw4l

tw4l commented Aug 26, 2026

Copy link
Copy Markdown
Member Author

Making sense to me. Since this has changed a model, I assume we need a corresponding migration?

Normally we would but this model isn't serialized to the database, just used to corral data from the k8s crawl job into a format that's easier for us to work with.

@tw4l tw4l modified the milestone: 1.26 Release Sep 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Seed file presigned URL expires on long-running resumed crawls

2 participants