Skip to content

fix: shorten interrupted task redelivery - #276

Merged
suguanYang merged 1 commit into
mainfrom
fix/wangbinqi/shorten-worker-redelivery
Aug 13, 2026
Merged

fix: shorten interrupted task redelivery#276
suguanYang merged 1 commit into
mainfrom
fix/wangbinqi/shorten-worker-redelivery

Conversation

@suguanYang

Copy link
Copy Markdown
Contributor

Summary

  • reduce the Redis broker visibility timeout from 12 hours to 75 minutes
  • keep the task hard limit, visibility timeout, and stale-job expiry as one validated ordering
  • reject environment overrides that would expire a processing job before Redis can redeliver it
  • add worker contract coverage for the valid runtime values and invalid override

Why

A hard-stopped late-acknowledged task currently remains invisible for 12 hours, while the stale-job sweeper marks processing jobs failed after 4 hours. The terminal job gate then prevents the eventual redelivery from restarting the job. The new ordering is:

3600s task hard limit < 4500s visibility timeout < 14400s processing expiry

This keeps the visibility timeout above the maximum task runtime while ensuring hard-stop redelivery occurs before stale-job expiry. Redelivery restarts processing from the beginning; it is not checkpoint resume.

Verification

uv run --directory /home/suguan/github.com/ontosAI/knowhere pytest -q apps/worker/tests/contract/test_worker_shutdown_contract.py apps/worker/tests/contract/test_worker_bootstrap_contract.py
5 passed

uv run --directory /home/suguan/github.com/ontosAI/knowhere ruff check packages/shared-python/shared/core/config/app.py packages/shared-python/shared/core/config/celery.py packages/shared-python/shared/core/celery_app.py apps/worker/tests/contract/test_worker_shutdown_contract.py
All checks passed!

Related to Ontos-AI/knowhere-api-infra#22.

@suguanYang
suguanYang merged commit f7e6c51 into main Aug 13, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant