fix: split Harness name and version in persona analytics (CLI-1771) - #7162
Conversation
persona.agent fused Claude Code's name, version, and role into one string (e.g. claude-code_2-1-233_agent), fragmenting it into ~96 distinct values in analytics and making a simple "how many Claude Code sessions" query impossible. Replace the hand-ported detect-agent signature table with the upstream github.com/vercel/detect-agent module (13 -> 20 harnesses, also fixes a live OpenCode detection bug), and split any version-shaped suffix off the detected name into a new persona.agent_version field, canonicalising the remaining name fragment against the module's own vocabulary rather than a hand-written alias list. persona.agent now always reports "claude_code" for Claude Code regardless of detection path. Needs OSS/legal sign-off before merge: the upstream module ships no LICENSE file (see PR description).
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
|
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
go-licenses save hard-fails on github.com/vercel/detect-agent because its repo has no LICENSE/NOTICE/COPYING file, breaking make configure and CI. Ignore it from go-licenses' auto-detection and supply the Apache-2.0 text directly, matching the license package.json declares. Ref: CLI-1771
…snyk/cli into fix/CLI-1771-harness-name-version-split
This comment has been minimized.
This comment has been minimized.
robertolopezlopez
left a comment
There was a problem hiding this comment.
Last comment: probably, those posible corner cases in agent.go make agent_test.go really necessary after the fix
detect-agent's README recommends `name@version` for custom AI_AGENT declarations; versionSuffix only handled '_' and '/'. Also stop TestReport_Agent from leaking ambient agent-signature env vars (AI_AGENT, CLAUDECODE, etc.) from the shell running `go test` into subtests that don't declare them.
…split' into fix/CLI-1771-harness-name-version-split
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
Same ambient-env leak as persona_test.go's TestReport_Agent, in the two subtests of TestVercelDetectDoesNotNormalizeExplicitAIAgent that call detectagent.Detect() directly. Also switch the isolateEnv restore loop to t.Setenv, which golangci-lint's usetesting linter requires over raw os.Setenv (and fixes an unchecked error to boot).
…n-split' into fix/CLI-1771-harness-name-version-split
This comment has been minimized.
This comment has been minimized.
versionSuffix required 2+ numeric components to avoid matching a name
that merely ends in a digit, but that guard doesn't apply to '@': the
separator alone is the version signal per detect-agent's README,
whose own example ("devin@1") the old regex rejected. Split '@' into
its own atVersionSuffix pattern with a 1-component minimum; '_'/'/'
keep the 2-component guard since they can end an ordinary name.
This comment has been minimized.
This comment has been minimized.
Two separate compiled patterns plus a match-fallback dance was more machinery than the difference warranted. One regex now expresses both cases via alternation with two version groups (version1 for '_'/'/', version2 for '@'); SplitVersion just picks whichever fired. Behaviour unchanged, same test suite passes.
This comment has been minimized.
This comment has been minimized.
t.Setenv registers its own unset-cleanup; calling it from inside isolateEnv's t.Cleanup undid the restore right after performing it, leaving the caller's real environment empty instead of restored.
This comment has been minimized.
This comment has been minimized.
'@' is never part of a Harness name (detect-agent's own documented
convention), so it's unambiguous even when the fragment in front of
it isn't a known harness — unlike '_'/'/', which are ordinary
characters a real name can end with. SplitVersion("custom-agent@2.0")
now returns ("custom-agent", "2.0") instead of leaving it unsplit.
This comment has been minimized.
This comment has been minimized.
PR Reviewer Guide 🔍
|
Summary
persona.agentfused Claude Code's name, version, and role into one string (claude-code_2-1-233_agent), fragmenting it into ~101 distinct values in analytics and making "how many Claude Code sessions" impossible to query. This splits name and version into two fields.detect-agentsignature table (cliv2/internal/persona/agent) with the upstreamgithub.com/vercel/detect-agentmodule — grows coverage 13 → 20 harnesses and fixes a live bug where plain-CLI OpenCode was undetected (our port matched the launcher's env var, not the one OpenCode itself sets).name[_/]version[_/role]), and canonicalises the remaining fragment againstdetect-agent's ownKnownAgentsvocabulary rather than a hand-written alias list.persona.agent_versionis written only when a version is found;persona.agentalways resolves toclaude_codefor Claude Code regardless of whether it was detected viaAI_AGENTor via environment signature.cursor-cli,augment-cli, and every other already-clean harness are unaffected — no version-shaped suffix, so the regex doesn't match and the value passes through unchanged.Signature renames, flagging ahead of release per the spec: dropping the hand-ported table for upstream
detect-agent's own vocabulary changes the canonical string several harnesses report. None of these are remapped back to the old values — intentional, current analytics data for these harnesses is already messed up, so this is a good time to adopt vercel's naming instead of preserving a value we hand-picked ourselves:DetectAgent()Detect()claudeclaude_codegeminigemini_clicodexcodex_cliopencodeopen_codeAI_AGENT=github-copilot-cligithub-copilotgithub-copilot-cliThis lands in the next release; will also be called out as a Jira comment on CLI-1771.
Datadog: where the new field lands
Confirmed against live
analytics-servicelogs (service:analytics-service "analytics payload"). Today:After this change,
agent_versionis a new sibling key at the same nesting level, queryable as@analytics-service.interaction.extension.persona.agent_version:agent_versionis omitted from the payload entirely (not an empty string) whenever no version is parsed out.License bundling — fixed
github.com/vercel/detect-agenthas noLICENSEfile in its repo (confirmed via the GitHub API), despite tagged releases and external contributors. This hard-failed CI and any local build.Every third-party Go dependency in
cliv2ships its license text embedded in the CLI binary:cliv2/scripts/prepare_licenses.gorunsgo-licenses save ./...and writes oneLICENSE(orNOTICE/COPYING) file per module path undercliv2/internal/embedded/_data/licenses/, whichcliv2/internal/embedded/file.gopulls into the binary via//go:embed _dataand surfaces to end users as the OSS notices. This step runs as part ofmake configureincliv2/Makefile, ahead ofmake build.Reproduced locally on this branch, matches CI exactly:
So
make configure/make buildfail outright with this dependency in place — this is not a merge that could accidentally ship unnoticed; the build gate stops it.go test ./...still passes (it doesn't go throughmake configure), which is why the test-plan checkboxes below are green despite this blocker.The script already has a manual-override mechanism for packages
go-licensescan't auto-detect (github.com/davecgh/go-spew,github.com/alexbrainman/sspi,github.com/pmezard/go-difflib,go.devare each fetched from a hardcoded URL) — butgo-licenses savefatal-exits before reaching that step for a package it finds zero license-like files for at all, so the override alone wasn't reachable. Fixed with two changes incliv2/scripts/prepare_licenses.go:--ignore github.com/vercel/detect-agenton thego-licenses saveinvocation, so it no longer requires a verdict on that package (dependencies of the ignored package would still be checked, but it has none).https://www.apache.org/licenses/LICENSE-2.0.txt), matchinggithub.com/vercel/detect-agent's ownpackage.jsondeclaration ("license": "Apache-2.0") — its repo just never committed the file.Verified both changes are needed: removing the
--ignoreand keeping only the manual-download entry still reproduces the exact same fatal error, sincego-licenses savenever reaches the manual-download loop. With both in place,go run scripts/prepare_licenses.goexits 0 and embeds a real Apache-2.0LICENSEfile forgithub.com/vercel/detect-agent.Note this asserts Apache-2.0 based on
package.json's declaration, not on a license file the copyright holder actually published — still worth an explicit legal nod given that.Test plan
go test ./...passes incliv2andcliv2-privatego vet ./...andgolangci-lint runclean on changed packagesgo mod tidy -diffclean in bothcliv2andcliv2-privatecliv2/internal/persona/persona_test.go(TestReport_Agent) at the single seam where persona values are recorded onto analytics — extends the existing fake-analytics-recorder test rather than a new harness, usingt.Setenvper case (the upstream module reads the process environment directly, no injection point). Covers every real-world shape observed in production: fused underscore (both_agentand_harnessroles), legacy slash format normalising to the same bucket, bare signature detection, an already-clean harness, a fused identifier with no version-shaped segment, an identifier that fails to canonicalise, a redacted (***) identifier, and a version-shaped run with no trailing role.analytics-servicelogs: no harness other than Claude Code currently reports a version-shapedpersona.agentvalue at any real volume, confirming the ticket's Out-of-Scope call.persona.agent-value count and confirm Claude Code's ~101 buckets collapse to one, withpersona.agent_versionpopulated on those events.Ref: CLI-1771