Skip to content

Increase graceful shutdown timeout for prometheus/prometheus2 jobs#596

Open
ybykov-a9s wants to merge 1 commit into
cloudfoundry:masterfrom
ybykov-a9s:fix/increase-prometheus-graceful-shutdown-timeout
Open

Increase graceful shutdown timeout for prometheus/prometheus2 jobs#596
ybykov-a9s wants to merge 1 commit into
cloudfoundry:masterfrom
ybykov-a9s:fix/increase-prometheus-graceful-shutdown-timeout

Conversation

@ybykov-a9s

Copy link
Copy Markdown
Contributor

I noticed that in some cases 25s given to prometheus to shut down using any kind of bosh stop might be not enough. In this case further logic runs kill -SIGKILL against prometheus process. It also has two downsides:

  • 25s + time used by monit and time used by bosh-agent together might overlap default bosh-agent timeout 30s, and bosh task fails in that case.
  • killing prometheus process with -SIGKILL leads to data loss on big deployments, because prometheus was not able to flush all WAL chunks from memory to the disk.
  • hard killing prometheus using -SIGKILL most likely would force prometheus to rebuild its WAL from scratch on the next boot.

There is no strict recommendations how much time prometheus might need for a proper graceful shutdown, especially in big environments.

This PR increases time for a graceful shutdown from 25s to 110s. It also configures bosh-agent to wait 10s more than graceful shutdown is configured before reporting that the job failed. Current leftover 5s length is not always enough for bosh-agent. Updating it to 10s should be enough to never give up in most cases.

The PR also makes the graceful shutdown timeout configurable.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Development

Successfully merging this pull request may close these issues.

1 participant