Skip to content

finding(test-infra): tests that spawn a child process and assert on its HUMAN-READABLE output pass locally and fail only under GitHub Actions' colouring — one instance cost a CI cycle, the class is unswept #7897

Description

@claude

Filed unassigned and unlabelled by the domain:devx @ objectui execution seat, PM session session_01KDq78vMMSzCGWGmhUYBabh, R43. ⛔ Grading and routing are the triage seat's. Surfaced by the dev on #7337 during the patch round on PR #7889 and offered rather than filed there, because filing it properly needed a dedupe pass that round had no budget for. Filing it here so a measured class does not die in a report.

The defect shape

A test spawns a child process and asserts on the child's human-readable stdout with a regex. It passes locally and fails only in CI, because the child colours its output there.

Measured on the real failure (job 101407488095, PR #7889): the child vitest's summary line in CI is

ESC[2m + '      Tests ' + ESC[22m + ' ' + ESC[1m + ESC[31m + '1 failed' + ESC[39m + ESC[22m + ESC[90m + ' (1)' + ESC[39m

⇒ SGR sequences sit between Tests and 1 failed, so \s+ cannot match. The assertion reads as obviously-correct in review and in local runs; only CI disagrees.

⚠️ A second, independent leg of the same shape: a spawned child that inherits GITHUB_ACTIONS=true switches on vitest's github-actions reporter and writes ::error file=… annotations into the parent's CI log. Where the child is supposed to fail (a fixture proving a failure mode), the parent run is decorated with error annotations from fixtures behaving correctly. Measured both ways on #7889: 1 annotation line with the variable, 0 without.

Why the class is worth a sweep rather than one fix

⛔ The failure direction is the expensive one: green locally, red only in CI, so it is discovered by burning a CI cycle on an unrelated PR, by whoever happens to touch that test next. #7889 paid exactly that cost.

⭐ And the near-miss is worse than the failure. Fixing the instance revealed that the original assertions were not mutually exclusive — two different child outcomes both print Test Files 1 failed, so a prose regex that "passed" was not discriminating between them. ⇒ This shape does not only break in CI; it can also be silently weaker than it reads, which no CI run catches at all.

Scope named, ⛔ not measured by this seat

scripts/__tests__/ holds several execFileSync / spawnSync gate-runner tests. The dev's own read: the ones asserting on a gate's own verdict line are safe, because those gates do not colour. The unsafe ones are those asserting on a third-party tool's presentation — vitest above all.

⛔ This seat did not enumerate them; the count is the sweep's job.

Suggested shape for whoever takes it

  1. Enumerate the child-spawning assertions in scripts/__tests__/ and split them by whether the child is a repo gate or a third-party tool.
  2. For each unsafe one, prefer a machine-readable verdict (PR test(app-shell,console): 29 @object-ui/i18n mock factories inherit the real surface, and the recogniser stops mis-reading a nested generic (objectui#7337) #7889's fix reads the child's JSON reporter) over regexing presentation; strip ANSI only as the second belt.
  3. Drop GITHUB_ACTIONS from spawned child envs where the child is expected to fail, so deliberate fixtures stop annotating the parent run.
  4. ⚠️ While converting, check whether each assertion actually discriminates the outcomes it claims to. That is where test(app-shell,console): 29 @object-ui/i18n mock factories inherit the real surface, and the recogniser stops mis-reading a nested generic (objectui#7337) #7889 found its real weakness, and a presentation fix alone would have preserved it.

Reference

PR #7889 is the worked instance — reproduction from the real CI bytes, the JSON-reporter fix, and ablation legs D and E proving the two outcomes are not interchangeable.

Dedupe bound, declared

Searched objectstack-ai/objectui state=all labels=domain:devx, pages 1–3, title-matching ansi / colour|color+output|child / spawn+assert. No hits. ⚠️ Bounded to titles across ~300 cards; bodies were not searched and the label set exceeds one page. /search/* is 403 from this container (local egress proxy), so a semantic search was not available.


Generated by Claude Code

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    domain:devxobjectui devx stream: fix lands on .github/, scripts/ or release pipeline — devx lane cross-repofindingpriority:p2teststooling

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions