test: drain all reloads before ConcurrentReloads asserts at rest - #1923
Merged
Conversation
ConcurrentReloadsDoNotCorruptTheChartSeries reddened CI ~1-in-4700 with `Assert.Single() Failure: The collection contained 2 items. Collection: [0, 3]`. Not a production bug and not a corrupt series — a test that read while a reload was still in flight. ReloadAsync applies its five ReplaceWith calls one at a time; between two of them a series is legitimately mid-clear (0) while another is already refilled (3). In the app the reader is the LiveCharts observer on the same UI thread as the reload, so that transient is never observed. The test resumed reloads on the thread pool and asserted while trailing ones ran. The bug was in the drain: the test fired 20 fire-and-forget reloads, awaited ONE command, then fired 20 MORE — and asserted. The 20 trailing reloads were still applying ReplaceWith when the assertion read the series. Now every reload is started through the command (its task collected), all are awaited via Task.WhenAll, and a final lone reload with nothing fired after it guarantees a coherent rest state. Both entry points (the SelectedRange changed-handler and the command) still overlap through the one gate, so the concurrency being exercised is unchanged. Reproduced and fixed deterministically off-tree: a spin-read right after the old `await refresh` observed a torn [0, n] series in 297/300 rounds; with the drain, 0/300. The single-read test hit that window ~1-in-thousands on loaded CI, which is why it surfaced as a rare red rather than a reliable one. No production change: adding a lock so a cross-thread reader never sees the transient would be dead complexity for a race the single-threaded-UI app cannot have.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The symptom
ResourceHistoryViewModelReloadTests.ConcurrentReloadsDoNotCorruptTheChartSeriesreddened CI~1-in-4700:
It passed on the PR that introduced its neighbourhood and on thousands of other runs; it failed once,
on an unrelated merge. That profile — rare, load-dependent, in a test whose name is literally about
concurrency — is the one this project has learned not to wave away as "flaky".
Root cause: a test that read mid-flight, not a product bug
ReloadAsyncapplies its fiveReplaceWithcalls one at a time (cpu, ram, gpu, then two tempseries). Between two of them a series is legitimately mid-clear — 0 points — while an earlier one is
already refilled to 3. At rest all five agree; mid-reload they don't.
[0, 3]is thattransient.
In the running app the reader is the LiveCharts observer on the same UI thread as the reload, so
it never observes the in-between state. A headless test resumes the reload on the thread pool, so a
cross-thread read can see it — and that is exactly what the test did:
It awaited a single reload while twenty more were still in flight, then read the series. The invariant
it checks ("all usage series equal length") only holds at rest, and the test never reached rest.
The fix (test only)
Route every reload through the command so its
Taskcan be collected, await them all, then one finallone reload with nothing fired after it — a guaranteed coherent rest state. Both entry points (the
SelectedRangechanged-handler and the command) still overlap through the one gate, so theconcurrency under test is unchanged; only the drain is added.
No production change. Adding a lock so a cross-thread reader never sees the transient would be dead
complexity for a race the single-threaded-UI app cannot have.
Verification (reproduced and fixed deterministically)
The single-read failure is ~1-in-thousands and would not reproduce in a warm isolated loop (7,200
runs, all green — no thread-pool pressure). So I demonstrated the mechanism deterministically
instead: after the old storm, spin-read the three series lengths right where the assertion sits.
[0, n]readThat 297/300 is the same window the single-read assertion lands in ~1-in-4700 on loaded CI. With the
drain the window is gone. The fixed test then ran 4,000× at 8-wide concurrency (mimicking xUnit's
parallel collections) with zero failures.
test:— no release. All four projects build 0/0; diff is one test file; leak scan clean.