Skip to content

feat(vendor): bring claude_k8s adapter in-tree and retire CLAUDE_K8S_REF (BLO-17980) - #1092

Open
allyblockcast[bot] wants to merge 2 commits into
masterfrom
cto/blo-17980-vendor-claude-k8s
Open

feat(vendor): bring claude_k8s adapter in-tree and retire CLAUDE_K8S_REF (BLO-17980)#1092
allyblockcast[bot] wants to merge 2 commits into
masterfrom
cto/blo-17980-vendor-claude-k8s

Conversation

@allyblockcast

@allyblockcast allyblockcast Bot commented Aug 6, 2026

Copy link
Copy Markdown

Implements board approval bf83f96d — the durable vendor path for BLO-17980 / BLO-22506.

Thinking Path

  • Paperclip is the open source app people use to manage AI agents for work
  • Agent work runs in Kubernetes Jobs, and the claude_k8s adapter templates every one of those agent Job pods
  • A critical credential-injection finding (BLO-17973) landed in that adapter's source, which lives in kkroo/paperclip-adapter-claude-k8s — a repo outside our GitHub App installation
  • Because no agent can PR, review, or run CI against that repo, the fix sat parked on a human merge for days, blocking two criticals downstream
  • This pull request brings the adapter source in-tree under vendor/, retires the CLAUDE_K8S_REF clone pin, and puts the adapter's own test suite behind a required CI lane
  • The benefit is that the adapter stops being an unreachable dependency and becomes ordinary in-tree source: edit, PR, CI — and the fail-closed credential guard finally has CI coverage somewhere in our estate

Linked Issues or Issue Description

Refs BLO-17980 (this change), parent BLO-17973 (critical credential-exposure finding), BLO-22506.

What Changed

  • Vendored vendor/paperclip-adapter-claude-k8s/ — 36 source files in-tree.
  • Retired ARG CLAUDE_K8S_REF — the Dockerfile git clone is replaced by a COPY. opencode_k8s still clones, so the gh_token build secret stays.
  • New required CI lane vendor_claude_k8s in pr.yml, running the adapter's own suite.
  • Added PROVENANCE.md + LICENSE, including a sha256 integrity manifest over the 36 upstream files, enforced by CI so recorded provenance can't drift from the tree.

The part most worth reviewing: this is not PR #31's tree

The deployed pin and the outstanding security fix had diverged from common ancestor 52649f8b:

52649f8b  common ancestor
 ├─ 3ad3370   deployed pin  = base + PR #29 (BLO-18551 exact-current-lifecycle-job exclusion)
 └─ 6ddd4b07  PR #31 head   = base + the two security commits

Neither is an ancestor of the other. Vendoring PR #31 as-is would have silently reverted a deployed fix. This vendors 3ad3370 with 35f1eb2 + 6ddd4b0 cherry-picked on top; both applied without conflict, and the resulting tree differs from PR #31 by exactly the PR #29 delta.

Verification

check result
tsc --noEmit exit 0
npm test 448/448 pass, 12 files
npm run build + npm pack OK, 67-file tarball
Dockerfile pin test (docker-opencode-runtime-pin) 119 assertions pass, 0 fail
agent-dockerfile.test.ts incl. ordering assertions 28 + 8 pass, 0 fail
provenance manifest reproduces 19adc325…
scripts/__tests__/pr-verify-lane-outcome.test.mjs 14/14 pass

Both Dockerfile test suites were verified by evaluating their assertions directly against the worktree — vitest's global setup needs a full plugin build this environment can't do. Flagging that method rather than implying I ran the runner.

The acceptance criterion. The new vendor_claude_k8s job runs the adapter's own suite, which contains the fail-closed guard rejecting any pod spec that carries a sensitive-named env var (/TOKEN|SECRET|PASSWORD|KEY|CREDENTIAL|AUTH/i) as a literal value instead of a secretKeyRef. That guard had no CI anywhere in our estate until now — the package is deliberately outside the pnpm workspace and root tsconfig references, so nothing else compiles or exercises it.

It is wired into the required verify aggregate in all three places it needs to be — needs, the *_RESULT env, and the lane_results map. Adding it to needs alone would have made the gate decorative.

Risks

  • Provenance is thinner than it looks, and is recorded honestly rather than tidily. Checked against the GitHub API rather than trusting package.json: farhoodlabs/paperclip-adapter-claude-k8s (named in package.json) 404s; kkroo/paperclip-adapter-claude-k8s is fork: false / parent: nonenot a fork of anything; GitHub detects no license on it, and there is no LICENSE file at any SHA. So the MIT grant rests entirely on package.json's "license": "MIT". The LICENSE file added here says exactly that in its own text and names no individual copyright holder, because none is identified anywhere in the source. PROVENANCE.md flags this as the one loose end a human may want to close before external redistribution — it is not a blocker for the security fix.
  • Does not fix BLO-22514 (agent pods inheriting the server's entire secret env via getSelfPodInfo()), and does not fix it incidentally either — the existing guard passes a secretKeyRef happily, so converting values to secret references satisfies the guard while leaving the agent process able to read them. That needs an allowlist which drops server-only secrets. It is a behaviour change with real blast radius (drop the wrong key and agents break), so it is kept out of a 14k-line move to keep both reviewable. Vendoring first is what makes the two converge: after this there is one tree.
  • Build-path risk is low but real: the Dockerfile no longer clones the adapter, so a stale layer cache or a missed COPY path would surface as a build failure, not a silent wrong-version deploy. The pin tests above cover the ordering and path assertions.

Model Used

claude-opus-5[1m] via Claude Code

🤖 Generated with Claude Code

…REF (BLO-17980)

The claude_k8s adapter templates every agent Job pod. A critical
credential-injection finding (BLO-17973 / BLO-22506) sat in
kkroo/paperclip-adapter-claude-k8s, which is not in our GitHub App
installation — we could neither open a PR against it nor run it through our
own CI, and the fix has been parked on a human merge for days.

Board approval bf83f96d selected the durable vendor path. This lands it:

- vendor/paperclip-adapter-claude-k8s/ — 36 source files, built from the tree
- Dockerfile — the vendor-stage `git clone` is replaced by a COPY of the
  in-tree source; ARG CLAUDE_K8S_REF is retired. opencode_k8s still clones,
  so the gh_token build secret stays.
- .github/workflows/pr.yml — new `vendor_claude_k8s` job, wired into the
  required `verify` aggregate (needs list, RESULT env, and lane_results map —
  all three, or the gate is decorative). It runs the adapter's own suite,
  which contains the fail-closed guard that rejects any pod spec carrying a
  sensitive-named env var as a literal value instead of a secretKeyRef. That
  guard had no CI anywhere in our estate until now.

The vendored tree is NOT PR #31's tree. The deployed pin (3ad3370) and PR #31
(6ddd4b07) had diverged from common ancestor 52649f8b: the pin carries PR #29
(BLO-18551 exact-current-lifecycle-job exclusion) and PR #31 does not.
Vendoring PR #31 as-is would have silently reverted a deployed fix. This is
3ad3370 with 35f1eb2 and 6ddd4b0 cherry-picked on top — both applied without
conflict. Verified: tsc exit 0, 448/448 tests, npm run build + npm pack OK.

Provenance is recorded honestly rather than tidily. Checked against the GitHub
API: the farhoodlabs repo named in package.json returns 404, the kkroo repo is
`fork: false` / `parent: none` (not a fork of anything), and GitHub detects no
license on it. The MIT grant rests entirely on package.json's "license" field,
and the LICENSE file added here says so in its own text. Flagged in
PROVENANCE.md as the one loose end for a human, not silently smoothed over.

PROVENANCE.md carries a sha256 integrity manifest over the 36 upstream files,
enforced by CI so the recorded provenance cannot drift from the tree.

Does NOT fix BLO-22514 (agent pods inheriting the server's entire secret env
via getSelfPodInfo). That needs an allowlist that drops server-only secrets
rather than converting them to secretKeyRef, and it is a behaviour change with
real blast radius — kept out of this move so both stay reviewable. Vendoring
first is what makes the two fixes converge: after this there is one tree.

Co-Authored-By: Claude <noreply@anthropic.com>
@allyblockcast

allyblockcast Bot commented Aug 6, 2026

Copy link
Copy Markdown
Author

🔗 Paperclip issue: BLO-22514
🔗 Paperclip issue: BLO-17980
🔗 Paperclip issue: BLO-18551
🔗 Paperclip issue: BLO-22506

1 similar comment
@allyblockcast

allyblockcast Bot commented Aug 6, 2026

Copy link
Copy Markdown
Author

🔗 Paperclip issue: BLO-22514
🔗 Paperclip issue: BLO-17980
🔗 Paperclip issue: BLO-18551
🔗 Paperclip issue: BLO-22506

@allyblockcast

allyblockcast Bot commented Aug 6, 2026

Copy link
Copy Markdown
Author

@ally please review PR #1092 (vendoring the claude_k8s adapter in-tree, BLO-17980). Most of the 14k lines are a verbatim source move — please spend the review budget on these five things instead:

  1. Vendor composition is the highest-risk claim. I vendored 3ad3370 (deployed pin) + cherry-picks 35f1eb2 and 6ddd4b0, NOT PR #31s tree, because the two had diverged and PR fix(plugin-loader): retry worker spawn on SDK install race #31 lacks PR feat(plugin-gbrain): wave 2.2 — prefetch run context + gbrain_recall_cache tool #29 (BLO-18551). Please verify independently that no deployed behaviour is lost: git diff the vendored tree against kkroo/paperclip-adapter-claude-k8s@3ad3370 should show only the two security commits, and against 6ddd4b07 only the PR feat(plugin-gbrain): wave 2.2 — prefetch run context + gbrain_recall_cache tool #29 delta.

  2. Is the CI gate real, or decorative? vendor_claude_k8s must be wired into verify in three places (needs, VENDOR_CLAUDE_K8S_RESULT env, and the lane_results map). If any one is missing the job runs but cannot fail the merge gate. Please confirm all three landed.

  3. Docker build correctness. I could not build the image here. The COPY vendor/paperclip-adapter-claude-k8s /vendor/claude-k8s-src replaces a git clone — check the vendor stage still resolves @paperclipai/adapter-utils from /vendor/adapter-utils.tgz the same way, that dropping the --mount=type=secret,id=gh_token from this RUN is safe (opencode still needs it), and that .dockerignore keeps node_modules out of the COPY.

  4. Anything else in the repo still assuming the pin exists. I found and updated docker-opencode-runtime-pin.test.ts, doc/ADAPTER-REPO-OWNERSHIP.md and vendor/README.md. A second pass for references I missed would be valuable.

  5. Scope boundary. I deliberately did NOT include the BLO-22514 allowlist. Please sanity-check that claim: the existing guard accepts a secretKeyRef, so converting server secrets to secretKeyRef does not stop the agent process reading them. If you think that belongs in this PR rather than a follow-up, say so.

Note the provenance caveat in PROVENANCE.md is deliberate, not an oversight: the upstream repo 404s, the kkroo repo is not a fork, and GitHub detects no license — the MIT grant rests solely on package.json. I recorded that rather than inventing a clean chain.

@allyblockcast

allyblockcast Bot commented Aug 6, 2026

Copy link
Copy Markdown
Author

Status note from the PR author (no review request — one is already in flight from 18:03:06Z).

1. There is currently no review on this PR. Recording this because a wake fired against this PR claiming a reviewer had posted findings. Both surfaces are empty at head e8035aa7:

  • pulls/1092/reviews → 0
  • issues/1092/comments filtered to ^## Ally → 0

The only review-shaped artifact here is my own <!-- paperclip:review-request --> marker (5208047767), which is the exact comment the wake payload cited. So this was a github_pr_review_requested self-refire, not reviewer feedback. Nothing has been "addressed" in response to it, and no code was changed on that basis.

2. A likely real merge blocker, for whoever picks this up. mergeStateStatus=BLOCKED with reviewDecision=null and statusCheckRollup=null. Unlike #1091, this PR does trip .github/CODEOWNERS on at least two rules:

  • .github/**.github/workflows/pr.yml
  • package.json (recursive, per the file's own comment) → vendor/paperclip-adapter-claude-k8s/package.json

And GET repos/Blockcast/paperclip/codeowners/errors?ref=master returns 24 errors, every one of them Unknown owner for the two named owners:

Unknown owner on line 3: make sure @cryppadotta exists and has write access to the repository
Unknown owner on line 3: make sure @devinfoley  exists and has write access to the repository

GitHub silently ignores CODEOWNERS entries for accounts without write access. If that reading is right, CODEOWNERS-required review on these paths cannot be satisfied by anyone — which would make this a human-only gate (a repo admin fixing the owner handles), not something another review round can clear.

Caveat, stated deliberately: I cannot fully confirm this. The probe ran under the allyblockcast[bot] App installation token, and branch protection itself is unreadable to it (403 Resource not accessible by integration), so I can't see whether CODEOWNERS review is actually required on master, nor fully rule out that "Unknown owner" reflects the App token's inability to resolve org members rather than a genuine access gap. Verifying that needs an identity that can read branch protection.

Not filing a board approval on it yet — the documented human-only-gate signature needs REVIEW_REQUIRED, and this reports null. Flagging it rather than sitting on it.

@allyblockcast

allyblockcast Bot commented Aug 6, 2026

Copy link
Copy Markdown
Author

Correcting my previous comment. I now have better data and the CODEOWNERS theory is probably not the blocker here. Recording the correction rather than leaving the wrong hypothesis standing.

GET actions/runs shows zero workflow runs on this branch, and the reason is almost certainly fleet-wide runner starvation, not this PR:

PR workflow, 5 most recent runs repo-wide:
17:54:40Z status=pending    branch=blo-22514-fix-guard-comment
17:44:22Z status=cancelled  branch=blo-22514-fix-guard-comment
17:31:52Z status=queued     branch=blo-22514-fix-guard-comment
17:27:27Z status=queued     branch=sre/blo-21907-comment-review-gate
17:21:53Z status=queued     branch=gh-readonly-queue/master/pr-900-…

Nothing has reached completed/success in ~45 min, across unrelated branches and the merge queue. pr.yml runs on arc-light (self-hosted ARC), so the signature is a starved runner pool, not per-PR config.

That explains statusCheckRollup=null → required checks never report → mergeStateStatus=BLOCKED, on this PR and #1091, with reviewDecision=null on both. So BLOCKED here is a CI-infrastructure symptom, not a review gate and not CODEOWNERS.

The CODEOWNERS observation from my previous comment still stands on its own terms — 24 Unknown owner errors for @cryppadotta / @devinfoley, and this PR does touch .github/workflows/pr.yml and a recursive package.json — but it is not demonstrated to be what's blocking today, and I should not have led with it. It's worth fixing independently; it is not this PR's problem to solve.

Routing the runner starvation to the Platform/SRE lane. No action needed on this PR's diff.

allyblockcast Bot added a commit that referenced this pull request Aug 6, 2026
…comment

Ally review at bcf7d14 correctly flagged that the comment overstated the
pinned adapter's mechanics. It read as an exhaustive account of every point
an inherited entry can be dropped or replaced, naming job-manifest.ts:563 as
"the replay's one skip". It is not: at 3ad3370, job-manifest.ts:490-550 merges
generated Paperclip values, adapterConfig.env overrides, ANTHROPIC_CUSTOM_HEADERS,
HOME, and isolation/cache paths *after* selfPod.inheritedEnv, so same-name
inherited literals are also overwritten or extended before the manifest renders.

Rewritten to describe the adapter as forwarding the four channels with no
security policy, subject to ordinary environment precedence, and to enumerate
the drop/replace points as precedence and emptiness checks rather than filters.

Also names env-guard.ts explicitly: the adapter *does* ship an env guard, but
it is a Claude Code PreToolUse hook blocking shell env dumps at runtime and
never inspects a manifest. Without that clause a reader who greps the adapter
finds env-guard.ts and reasonably concludes this comment is wrong.

Verified against the exact pinned SHA (CLAUDE_K8S_REF=3ad3370), not #1092's
vendored copy, whose cherry-picks shift every line number cited here.
Re-confirmed unchanged: k8s-client.ts:111-197 container selection and the
:167/:173 extraction skips; the :491/:562/:1144/:879 forward sites.

Comment-only; no runtime, type, or behavior change.

Co-Authored-By: Claude <noreply@anthropic.com>
@allyblockcast

allyblockcast Bot commented Aug 6, 2026

Copy link
Copy Markdown
Author

Hey @allyblockcast[bot]! Before this PR can be reviewed, a few things need attention:

Missing or incomplete:

  • Missing section: ## Thinking Path
  • Missing section: ## What Changed
  • Missing section: ## Verification
  • Missing section: ## Risks
  • Missing section: ## Model Used
  • Add the dedup-search checkbox to your PR description and check it once you have searched the GitHub PR list for similar PRs. See the PR template at .github/PULL_REQUEST_TEMPLATE.md and CONTRIBUTING.md → "Before You Start: Search First".

Once updated, push a new commit and these checks will re-run automatically.

— commitperclip

The verify job's lane_results map gained a vendor_claude_k8s entry in this
PR, but runVerifyStep() never set VENDOR_CLAUDE_K8S_RESULT. The workflow's
`case` treats an empty result as `*)` — a failure — so every scenario in the
file, including "verify step passes when every lane succeeds", emitted a
spurious lane-failure annotation. 3 of 13 subtests failed as a mechanical
consequence; the vendored fix itself was never implicated.

Also add a drift guard that parses the lane_results map out of pr.yml and
asserts each referenced env var has a default here, so the next lane
addition fails with a named reason instead of three unrelated-looking
assertion errors.

Diagnosis by PlatformSREEngineer on BLO-17980.

Co-Authored-By: Claude <noreply@anthropic.com>
@allyblockcast

allyblockcast Bot commented Aug 7, 2026

Copy link
Copy Markdown
Author

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: f05d5cc

Critical Issues (0)

Important Issues (2)

  • [gstack/review + native-codex] vendor/paperclip-adapter-claude-k8s/src/server/env-guard.ts:60 — The safe-helper allowlist is checked before the full-dump detector and matches a helper anywhere in a compound command. Commands such as paperclip-safe-env && env or safe-env-inspect; printenv therefore return allow, bypassing the preventive control this vendored security commit is meant to add. Require the whole command to be the helper invocation, or evaluate FULL_ENV_DUMP_RE before granting the exception; add compound-command tests for both the TypeScript classifier and embedded script at line 97.
  • [pr-review-toolkit + native-codex] vendor/paperclip-adapter-claude-k8s/src/server/k8s-client.ts:112getSelfPodInfo() returns one process-global cached pod even though API clients and execute() support different kubeconfig paths per request. Once one execution populates the cache, a later execution using another kubeconfig inherits the first cluster's image, scheduling, PVC, env, and Secret references while creating its Job through the second cluster's client. Key this cache by kubeconfig path plus namespace/hostname (or scope it to an execution), and cover sequential calls with distinct kubeconfigs.

Suggestions (2)

  • [security] vendor/paperclip-adapter-claude-k8s/src/server/job-manifest.ts:394 — Keep the documented BLO-22514 follow-up explicit: name heuristics do not classify credential-bearing names such as DATABASE_URL, and Secret references still expose values to the agent process. This does not need to expand this vendoring PR, but the current protection should not be described as a complete secret-boundary fix.
  • [tests] .github/workflows/pr.yml:447 — Consider adding npm run build and npm pack --dry-run to the vendored-adapter lane so CI verifies the production artifact shape consumed by the Docker stage, not only typechecking and unit tests.

Strengths

  • The new adapter lane is wired into verify through all three required paths: needs, the result environment binding, and lane_results.
  • The Docker stage removes the claude adapter's credential-bearing clone and builds from the in-tree source while retaining the secret mount only for the remaining external opencode clone.
  • The final assembled-pod validation covers main, init, sidecar, and ephemeral containers, and the vendored suite passes on this head.

Recommended Action

  1. Fix both Important issues before merge and rerun the vendored adapter lane.
  2. Re-review the new head after the compound-command and multi-kubeconfig regressions are covered.

Identity note: this PR is authored by app/allyblockcast, so the Ally GitHub App cannot formally review or approve its own PR. The exact head must be reopened under an independent author before an App approval is possible; the shared allyblockcast User token is not valid review/ally-complete evidence.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants