Skip to content

attempt to ensure metacontroller finalizers always run - #3598

Merged
ikreymer merged 6 commits into
mainfrom
issue-3365-increase-ttl-after-fin-add-finalizers
Aug 24, 2026
Merged

attempt to ensure metacontroller finalizers always run#3598
ikreymer merged 6 commits into
mainfrom
issue-3365-increase-ttl-after-fin-add-finalizers

Conversation

@ikreymer

@ikreymer ikreymer commented Aug 22, 2026

Copy link
Copy Markdown
Member

Looking into this issue more, it seems the ttlSecondsAfterFinished set to 0 is likely the main culprit for finalizers being skipped for these resources, as the job is cleaned up right away. However, to guarantee the finalizers are always run, they should also declared as part of the object metadata at creation time for the resources that have these hooks.

This PR:

  • set ttlSecondsAfterFinished to 10 for replica jobs, 1 for other background jobs (to avoid breaking tests which wait for these jobs to be finalized)
  • explicitly declares the appropriate metacontroller finalizer hook on each resource that has a finalize hook (crawljob, collindex, background-job) at resource creation time

Should fix #3365

- declare finalizers on each resource that has a finalize hook at creation time (crawljob, collindex, background-job)
- set ttlSecondsAfterFinished to 30 instead of 0 on background-job and create-replica background-job
- should fix #3365

The ttlSecondsAfterFinished set to 0 is likely the main culprit for finalizers being skipped for these resources, as the job is cleaned up
right away. However, to guarantee the finalizers are always run, they are also declared as part of the object metadata at creation time for
the resources that have these hooks.
@ikreymer
ikreymer requested a review from tw4l August 22, 2026 08:04
@ikreymer

Copy link
Copy Markdown
Member Author

Set the ttl lower for replica and other background jobs, as some code paths (like collection update) does wait for the jobs to finish, in tests. Perhaps keeping ttl at 0 or 1 does make sense, as long as the finalizer is set on create, as the whole point is to respond quickly as soon as the job is finished

@tw4l tw4l left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not convinced ttlSecondsAfterFinished is playing a part here (though I'm also fine with the small changes to those values here, just haven't seen evidence that plays into the bug), but explicitly setting the finalizers seems like a good step.

One suggestion where I think we need another finalizer added, and will need to do the same in #3577.

Comment thread chart/app-templates/background_job.yaml
@ikreymer

ikreymer commented Aug 24, 2026

Copy link
Copy Markdown
Member Author

I'm not convinced ttlSecondsAfterFinished is playing a part here (though I'm also fine with the small changes to those values here, just haven't seen evidence that plays into the bug), but explicitly setting the finalizers seems like a good step.

I think the finalizers are the real fix, so maybe no need to change the ttl, though it probably makes it more likely to happen. Removed the ttl changes for now so we can check if the finalizers definitively fix this.

Comment thread backend/test_nightly/test_delete_crawls.py
@ikreymer ikreymer added this to the 1.25 Release milestone Aug 24, 2026
Comment thread chart/app-templates/background_cron_job.yaml
Comment thread chart/app-templates/replica_deletion_cron_job.yaml
@ikreymer
ikreymer merged commit 394eeb9 into main Aug 24, 2026
24 checks passed
@ikreymer
ikreymer deleted the issue-3365-increase-ttl-after-fin-add-finalizers branch August 24, 2026 19:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Database is never updated for some background jobs after completion - jobs deleted before finalizer

2 participants