test(site-memory): stop the suite flaking on Windows git handles and timer skew - #484
Closed
Agnik47 wants to merge 1 commit into
Closed
test(site-memory): stop the suite flaking on Windows git handles and timer skew#484Agnik47 wants to merge 1 commit into
Agnik47 wants to merge 1 commit into
Conversation
`Unit tests (windows-latest, shard 1/2)` fails intermittently on unrelated
branches. The reported failures are always the same shape:
FAIL src/site-memory/checkpoint.test.ts > checkpoint compare-and-swap > …
Error: Test timed out in 5000ms.
Error: EBUSY: resource busy or locked, rmdir '…\webcmd-checkpoint-t05DZ4\.webcmd\sites'
The two errors are one cascade, not two problems. Each of these tests
spawns several real git processes against a throwaway home. That costs a
second or two idle and several times more under a parallel suite on
Windows, where process creation is expensive, so the test overruns
vitest's 5s default and is aborted mid-git. `afterEach` then removes the
temp directory while a git child is still holding `.git`, and the
recursive remove fails too — reported against the same test, and looking
like a filesystem bug rather than a clock.
Size the budget to what these tests actually do, and make cleanup
tolerate the handles. `GIT_TEST_TIMEOUT_MS` is 20s, the value the slowest
tests in these files had already opted into individually; applying it per
file makes the whole class consistent. It does not hide a hang — a test
that never finishes still fails, at 20s. `removeTempDirs()` passes
`maxRetries`/`retryDelay` to `rm`, which retries EBUSY/EPERM/ENOTEMPTY
for exactly this reason but does not by default.
Both live in one support module so the reasoning and the numbers have a
single home, and the eight site-memory test files that drive git use it.
Verified on Windows: before, every full-suite run reproduced the CI
failure locally, including the identical `EBUSY … \.webcmd\sites`. After,
three consecutive full-suite runs show no site-memory git failures. A
scratch test that sleeps 6s confirms the file-level budget takes effect,
which it would not under the 5s default.
Contributor
🟢 No documentation gap found — medium confidenceThe automated review found no documentation gap in the supplied changes. This review is advisory and does not block merging. |
Collaborator
|
cherry picked the commit so closing this |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two
src/site-memorytests fail in CI for reasons that have nothing to do with what they assert. Both are environmental, both land on unrelated branches, and between them they have reddenedmainand several open PRs.1. Windows: git handles outlive the test
Unit tests (windows-latest, shard 1/2)fails intermittently. It hit #481 on 3 Sep and #483 on 4 Sep, and that shard has gone red onmainseveral times recently. The failure is always the same shape:The two errors are one cascade, not two problems:
gitprocesses against a throwaway home. That costs a second or two on an idle machine and several times more under a parallel suite on Windows, where process creation is expensive — past vitest's 5s default.afterEachthen removes the temp directory while a git child still holds.git, so the recursive remove fails too — reported against the same test, and looking like a filesystem bug rather than a clock.I reproduced both locally on Windows, including the identical
EBUSY … \.webcmd\sitespath. The tests pass in isolation (checkpoint.test.ts: 48 tests, 43s) and only fail under full-suite parallelism, which matches CI exactly.Fix — size the budget to what these tests do, and let cleanup tolerate the handles. Both live in one support module,
__fixtures__/git-test-support.ts, so the reasoning and the numbers have a single home:GIT_TEST_TIMEOUT_MS = 20_000, applied per file viavi.setConfig. Not a new policy — it is the value the slowest tests in these same files had already opted into individually (}, 20_000)); applying it per file makes the class consistent instead of whack-a-mole. It does not hide a hang: a test that never finishes still fails, at 20s.removeTempDirs()passesmaxRetries/retryDelaytorm. Node retriesEBUSY/EPERM/ENOTEMPTYfor exactly this case, butmaxRetriesdefaults to 0.Applied to the eight site-memory test files that drive real git. No production code changes — the
EBUSYis a test-cleanup artifact, not a product defect. I checkedfile-lock.tsfor a leaked descriptor before concluding that; it closes its handle in afinally, so retrying is not papering over a leak.2. Any platform: the seed lookup budget is asserted to the millisecond
The first push of this PR went green on both Windows shards and then failed on
Unit tests (ubuntu-latest, shard 2/2):aborts after two seconds and does not retryasserts that a lookup guarded byAbortSignal.timeout(2000)took at least 2000ms byDate.now(). Node can run the timer callback when only 1999ms have elapsed by that clock — libuv compares against a loop time it caches and truncates to whole milliseconds, so the two disagree by up to a millisecond.Fix — the bound exists to show the lookup waited for the timeout instead of returning early, which a millisecond of slack still shows. The upper bound that proves it did not retry is unchanged.
Verification
On Windows 11, Node 22:
checkpoint.test.tsandself-learning.integration.test.tstiming out at 5000ms and then failing cleanup with the sameEBUSY … \.webcmd\sites.EPERM: symlinktests, which need Developer Mode and fail identically before this change.vi.setConfig({ testTimeout: GIT_TEST_TIMEOUT_MS }), confirming the file-level budget takes effect — it would fail at the 5s default.Unit tests (windows-latest, shard 1/2), the job that had been failing, passed.npm run typecheckclean.Notes
}, 20_000)markers left in these files are now redundant with the file default. I left them rather than widen the diff; happy to strip them if you'd prefer.src/hosted/programmatic-differential.test.ts(5s timeout) andsrc/browser/run/runner.test.ts(EBUSYunlinkingchrome_debug.log).