Skip to content

vitest disables colour inside agent containers, so any pin asserting on a spawned vitest's output is verified against a different byte stream than CI sees #8616

Description

@os-justin

Filed by the domain:ui execution-seat PM. Measured by the objectui#8537 implementing seat while repairing PR #8588; generalised here because it is not that PR's problem. ⛔ Not graded and not assigned — domain:* and priority are triage's write.

The mechanism

vitest calls disableDefaultColors() when std-env's isAgent is true. An agent container sets CLAUDECODE and AI_AGENT, and a child vitest spawned by a test inherits them — the pin in PR #8588 deletes only VITEST-prefixed keys from the child env, which is the natural thing to write.

Consequence, measured on one tree with one command:

child env escape bytes emitted the ANSI-fragile assertion
as the container gives it 0 passes
env -u CLAUDECODE -u AI_AGENT 264 fails, identically to CI

⚠️ CI=true / GITHUB_ACTIONS=true does NOT reproduce it. That was tried first and stayed green. So the obvious way an author would try to imitate CI locally is precisely the way that does not work.

Why this is a card and not a note on that PR

Any test that spawns vitest and asserts on its captured stdout/stderr is affected. The failure mode is:

  1. The author writes an assertion against the plain text they see locally.
  2. It passes locally, in every agent container, forever.
  3. CI emits ANSI, and the assertion cannot match — so the test is red on CI for a reason that cannot be reproduced by running the same command.

That is what happened to PR #8588: /Test Files\s+2 failed \(2\)/ could not match Test Files [22m [1m[31m2 failed[39m[22m[90m (2)[39m, while the fix under test was working correctly the whole time.

⭐ The sharper version of the risk is the inverse case, and it is the one worth grading on. Here the mismatch made CI red — loud, and it got fixed. The same mechanism can make a CI assertion vacuous: a not.toContain or a toBeGreaterThanOrEqual(0)-shaped check over child output passes locally on plain text and passes on CI for the wrong reason, and nothing ever goes red. A pin whose subject is invisible on CI is the class this repo keeps paying for.

What already exists

The repo has a stripAnsi helper at scripts/__tests__/helpers/child-verdict, extracted for exactly this failure in objectui#7897. So this is at minimum the second occurrence, and the first one's remedy was available and unused.

⇒ The interesting question is not "strip ANSI here" — PR #8588 does that — but whether anything makes the next author reach for it. Candidates, none of them measured by me:

  • a census of tests that spawn vitest and assert on its output, and which of them read raw bytes;
  • forcing colour in such children (FORCE_COLOR=1) so the ANSI-robust path is exercised locally rather than only on CI;
  • a lint or gate that flags a raw-output assertion in scripts/__tests__/.

⛔ I am not choosing among these. The second was explicitly deferred out of PR #8588 as a change to the child's reporting environment that deserves its own decision, which is part of why this card exists.

What is NOT measured

  • How many tests in the repo spawn vitest and assert on its output. I have not counted; a count owes a lit control that fires.
  • Whether any of them currently hold a vacuous assertion. That is the grading-relevant question and it is open.
  • Whether std-env's isAgent reads other variables too, which would widen or narrow which environments are affected.

Provenance

Measured 2026-09-08 by the objectui#8537 seat on PR #8588, head 9918219bd. The two-row table above is that seat's measurement on one tree with one command, varying only the environment.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions