Increase graceful shutdown timeout for prometheus/prometheus2 jobs#596
Open
ybykov-a9s wants to merge 1 commit into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I noticed that in some cases 25s given to
prometheusto shut down using any kind ofbosh stopmight be not enough. In this case further logic runskill -SIGKILLagainstprometheusprocess. It also has two downsides:monitand time used bybosh-agenttogether might overlap defaultbosh-agenttimeout 30s, and bosh task fails in that case.prometheusprocess with -SIGKILL leads to data loss on big deployments, becauseprometheuswas not able to flush all WAL chunks from memory to the disk.prometheususing -SIGKILL most likely would forceprometheusto rebuild its WAL from scratch on the next boot.There is no strict recommendations how much time
prometheusmight need for a proper graceful shutdown, especially in big environments.terminationGracePeriodSecondsto 600s.This PR increases time for a graceful shutdown from 25s to 110s. It also configures
bosh-agentto wait 10s more than graceful shutdown is configured before reporting that the job failed. Current leftover 5s length is not always enough for bosh-agent. Updating it to 10s should be enough to never give up in most cases.The PR also makes the graceful shutdown timeout configurable.