Give mezod time to shut down cleanly in all variants - #101
Merged
Conversation
mezod must flush state and close its LevelDB databases on shutdown. When the process is killed before it finishes, the databases can be left with a corrupted MANIFEST (observed as 'leveldb: manifest corrupted (field comparer): missing' on the next start), which makes the node unable to start and requires a resync or restore. The supervisor defaults are too tight for that: Kubernetes gives pods 30 seconds, docker compose 10 seconds, and systemd 90 seconds before escalating to SIGKILL. Set an explicit 120-second grace period in all three deployment variants: - Helm chart: new terminationGracePeriodSeconds value (default 120) wired into the StatefulSet pod spec; chart version bumped to 12.1.0 - docker compose: stop_grace_period on the mezod service - native systemd unit: TimeoutStopSec A clean shutdown normally takes seconds, so the longer grace has no effect in normal operation; it only delays SIGKILL when shutdown is genuinely slow or stuck.
9 tasks
The pre-commit helm-docs-built hook sorts the values table in file order (--sort-values-order=file), but the committed README was in alphanumeric order and out of date, so CI failed. Regenerate it so the table matches values.yaml order and includes terminationGracePeriodSeconds.
lukasz-zimnoch
force-pushed
the
raise-shutdown-grace-period
branch
from
July 7, 2026 11:03
9e58a40 to
61ee900
Compare
tomaszslabon
approved these changes
Jul 7, 2026
lukasz-zimnoch
marked this pull request as ready for review
July 7, 2026 11:57
tomaszslabon
added a commit
to mezo-org/mezod
that referenced
this pull request
Jul 7, 2026
Depends on: mezo-org/validator-kit#101 ### Introduction A mezod node killed before it finishes shutting down can be left with a corrupted LevelDB MANIFEST (`leveldb: manifest corrupted (field 'comparer'): missing` on the next start), which makes the node unable to start without a restore. This happened to `mezo-node-3` on the testnet after an unclean stop, and a mainnet validator operator reported the same failure signature independently. The pods currently run with the implicit Kubernetes default of 30 seconds before SIGKILL, which is too tight for a node that must flush state and close five databases on shutdown. Chart `mezod` 12.1.0 (mezo-org/validator-kit#101) sets `terminationGracePeriodSeconds: 120` on the node pods by default. ### Changes Bump the `mezo-org/mezod` chart pin from 12.0.1 to 12.1.0 in both helmfiles, covering all five testnet validators (`mezo-staging`) and the mainnet RPC node (`mezo-production`). ### Testing Both helmfiles parse as valid YAML. `helmfile template` cannot be run until the 12.1.0 chart tag is published from mezo-org/validator-kit#101; the chart itself passes `helm lint` and renders `terminationGracePeriodSeconds: 120` in the StatefulSet pod spec there. Keep this PR as draft until 12.1.0 is published, then roll out with `helmfile apply` per cluster. --- ### Author's checklist - [x] Provided the appropriate description of the pull request - [x] Updated relevant unit and integration tests - [x] Updated relevant documentation (`docs/`) or specification (`x/<module>/spec/`) - [x] Assigned myself in the `Assignees` field - [x] Assigned `mezod-developers` in the `Reviewers` field and notified them on Discord ### Reviewer's checklist - [ ] Confirmed all author's checklist items have been addressed - [ ] Considered security implications of the code changes - [ ] Considered performance implications of the code changes - [ ] Tested the changes and summarized covered scenarios and results in a comment
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Introduction
A mezod node killed before it finishes shutting down can be left with a corrupted LevelDB MANIFEST (
leveldb: manifest corrupted (field 'comparer'): missingon the next start), which makes the node unable to start and requires a restore or resync to recover. This was observed on a testnet node and reported independently by a mainnet validator operator, in both cases after an unclean stop. The supervisor defaults in all deployment variants are too tight for a node that needs to flush state and close five databases on shutdown: Kubernetes gives pods 30 seconds, docker compose 10 seconds, and systemd 90 seconds before escalating to SIGKILL.Changes
All three deployment variants now give mezod an explicit 120-second shutdown grace period:
terminationGracePeriodSecondsvalue (default120) wired into the StatefulSet pod spec. Chart version bumped to12.1.0; README regenerated with helm-docs.stop_grace_period: 2mon themezodservice.TimeoutStopSec=120in the generatedmezo.serviceunit.A clean shutdown normally completes in seconds, so the longer grace period has no effect in normal operation; it only delays SIGKILL when a shutdown is genuinely slow or stuck.
Testing
helm lintpasses andhelm templaterendersterminationGracePeriodSeconds: 120in the StatefulSet pod spec.docker compose configparses the compose file and rendersstop_grace_period: 2m0son themezodservice.bash -n/sh -npass onnative/v-kit.sh.