ci: add a crash consistency test - #7400
Merged
Merged
Conversation
Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com>
solracsf
force-pushed
the
test/crash-consistency
branch
from
August 15, 2026 15:12
cc371df to
bb818f9
Compare
Xuyuchao-juice
approved these changes
Aug 17, 2026
Move the crash consistency scripts under .github/scripts/command/ and run them from a new step in command.yml instead of a standalone workflow, so the test follows command's trigger rules and meta matrix (mysql/redis/tikv). Drop the separate crash-consistency.yml. Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
60s can elapse under a loaded runner and let staging drain before the kill, defeating the replay check. 1h guarantees fsynced blocks are still in staging when the mount is killed. Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Ensure CI cleanup collects mount PIDs before unmounting so prepare_test waits for the old JuiceFS mount process to exit before flush_meta drops and recreates SQL metadata. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Nothing currently checks that a mount which dies without unmounting keeps the data it said was durable. The chaos workflow runs vdbench under injected faults, but the configs set no data validation and the error check on its log is commented out, so a fault never fails the run. The kill -9 calls under
.github/scripts/commandkill a CLI command, not the mount.The writer records a file only after
fsync()on the file andfsync()on its parent directory both returned, into a manifest kept outside the mount and fsync'd per entry. Every entry is therefore a promise POSIX requires to survive a crash, and files that were written but not yet fsync'd are ignored, since losing those is allowed. The mount is killed with SIGKILL mid-write, remounted, and every recorded file is checked by size and sha256, followed by fsck for metadata pointing at objects that were never uploaded.Each test then re-reads everything through an empty cache directory, so the blocks come from object storage rather than the local cache that survived the kill.
The
--writebacktest mounts with--upload-delayso that blocks acked by fsync are still sitting in the staging directory when the kill lands, which is what exercises the staging replay on the next mount. Without the delay the uploads keep up with the writer and the kill finds staging empty, so the run passes without covering the replay at all; the test fails instead of passing quietly in that case.The mount process is matched by its mount point: CI runs a second juicefs mount for the coverage directory, and killing that one would leave the filesystem under test healthy.