CI failure details
The job failed on the first attempt; this is not a matrix-cancellation artifact. The failed job completed in the same minute as the Slack alert.
Race detector evidence
The full race report identifies an unsynchronized write to the aiTaskStoreWrapper callback in TestResumeTask/Build_Internal_Error, concurrent with a background workspace-build orchestrator transaction:
WARNING: DATA RACE
Write at 0x00c1700ba818 by goroutine 241256:
github.com/coder/coder/v2/coderd_test.TestResumeTask.func13()
/home/runner/work/coder/coder/coderd/aitasks_test.go:3405 +0xc71
github.com/coder/coder/v2/coderd/database.(*sqlQuerier).InTx()
/home/runner/work/coder/coder/coderd/database/db.go:181 +0x1f2
github.com/coder/coder/v2/coderd_test.aiTaskStoreWrapper.InTx()
/home/runner/work/coder/coder/coderd/aitasks_test.go:135 +0x15d
...
Previous read at 0x00c1700ba818 by goroutine 943001:
github.com/coder/coder/v2/coderd_test.(*aiTaskStoreWrapper).InTx()
<autogenerated>:1 +0x44
github.com/coder/coder/v2/coderd/database/dbauthz.(*querier).InTx()
/home/runner/work/coder/coder/coderd/database/dbauthz/dbauthz.go:1301 +0x106
github.com/coder/coder/v2/coderd/wsbuildorchestrator.(*Orchestrator).processNext()
/home/runner/work/coder/coder/coderd/wsbuildorchestrator/wsbuildorchestrator.go:239 +0x50a
github.com/coder/coder/v2/coderd/wsbuildorchestrator.(*Orchestrator).processAll()
/home/runner/work/coder/coder/coderd/wsbuildorchestrator/wsbuildorchestrator.go:214 +0x145
github.com/coder/coder/v2/coderd/wsbuildorchestrator.(*Orchestrator).run()
/home/runner/work/coder/coder/coderd/wsbuildorchestrator/wsbuildorchestrator.go:196 +0x15c
The directly affected subtest then reports:
=== FAIL: coderd TestResumeTask/Build_Internal_Error (0.88s)
testing.go:1712: race detected during execution of test
The package summary showed 133 failures, but those are race-detector fallout after this race tripped; many otherwise-passing tests were marked failed. No panic, OOM, killed process, or resource-exhaustion evidence was present.
Root cause assessment
Classification: Data race.
TestResumeTask/Build_Internal_Error passes &wrapped into coderdtest.NewWithAPI, starting background components that retain the store. After waiting for the pause build, the test mutates the shared field without synchronization:
wrapped.insertWorkspaceBuild = func(ctx context.Context, arg database.InsertWorkspaceBuildParams) error {
return xerrors.New("insert failed")
}At the same time, wsbuildorchestrator.processNext calls o.db.InTx. The wrapper's value-receiver InTx copies/reads insertWorkspaceBuild, racing with the assignment above. This is a test-code concurrency defect, not infrastructure and not a failure introduced by the CI commit.
A likely fix is to keep the callback immutable after server startup and gate its behavior with synchronized state, or add a mutex/atomic-backed setter and synchronized reads in the wrapper.
Assignment analysis
Commands used for the failing test area:
git blame -L 3368,3411 coderd/aitasks_test.go
git log --oneline -10 --follow coderd/aitasks_test.go
Repository history confirms commit 47b8ca94 introduced TestResumeTask, the Build Internal Error subtest, and the exact unsynchronized callback assignment together. That commit is by Sas Swart (SasSwart). Later changes to the file either touched other subtests or unrelated areas. Assignment therefore follows the specific failing test-function ownership, not the author of the failing CI commit.
Duplicate search
No existing issue matched TestResumeTask, Build Internal Error, aitasks_test.go:3405, aiTaskStoreWrapper, insertWorkspaceBuild, or the wsbuildorchestrator.processNext race stack.
Related but different broad race-detector reports:
Reproduction
go test ./coderd -race -run 'TestResumeTask/Build_Internal_Error' -count=100
The full-package race run may make the background-orchestrator timing more reproducible.
CI failure details
test-go-race-pg)76eb43a78e459905b8c6d618fee5526245e3dc0dThe job failed on the first attempt; this is not a matrix-cancellation artifact. The failed job completed in the same minute as the Slack alert.
Race detector evidence
The full race report identifies an unsynchronized write to the
aiTaskStoreWrappercallback inTestResumeTask/Build_Internal_Error, concurrent with a background workspace-build orchestrator transaction:The directly affected subtest then reports:
The package summary showed
133 failures, but those are race-detector fallout after this race tripped; many otherwise-passing tests were marked failed. No panic, OOM, killed process, or resource-exhaustion evidence was present.Root cause assessment
Classification: Data race.
TestResumeTask/Build_Internal_Errorpasses&wrappedintocoderdtest.NewWithAPI, starting background components that retain the store. After waiting for the pause build, the test mutates the shared field without synchronization:At the same time,
wsbuildorchestrator.processNextcallso.db.InTx. The wrapper's value-receiverInTxcopies/readsinsertWorkspaceBuild, racing with the assignment above. This is a test-code concurrency defect, not infrastructure and not a failure introduced by the CI commit.A likely fix is to keep the callback immutable after server startup and gate its behavior with synchronized state, or add a mutex/atomic-backed setter and synchronized reads in the wrapper.
Assignment analysis
Commands used for the failing test area:
Repository history confirms commit
47b8ca94introducedTestResumeTask, theBuild Internal Errorsubtest, and the exact unsynchronized callback assignment together. That commit is by Sas Swart (SasSwart). Later changes to the file either touched other subtests or unrelated areas. Assignment therefore follows the specific failing test-function ownership, not the author of the failing CI commit.Duplicate search
No existing issue matched
TestResumeTask,Build Internal Error,aitasks_test.go:3405,aiTaskStoreWrapper,insertWorkspaceBuild, or thewsbuildorchestrator.processNextrace stack.Related but different broad race-detector reports:
TestResumeTaskoccurrenceTestPatchChat/TestProvisionerJobsrace failures; no matching root-cause stackReproduction
The full-package race run may make the background-orchestrator timing more reproducible.