feat(k8s): cluster-wide cpu budget env, and events RBAC for the pod-warning watcher - #1184
feat(k8s): cluster-wide cpu budget env, and events RBAC for the pod-warning watcher#1184PatrickKalkman wants to merge 1 commit into
Conversation
…-warning watcher WARREN_K8S_CPU_REQUEST_MILLICORES / _LIMIT_MILLICORES mirror the memory knob: a run pod on a two-core node never scheduled the 1-CPU default request, and the repo cannot always carry a .warren/ resources block. The base Role never granted list/watch on core Events, so the pod-warning watcher (warren-32f8) reconnected on 403 forever. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rf5tbgdUDmHzKnNXpiJLvg
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 830e5aa345
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| resources: ["events"] | ||
| verbs: ["get", "list", "watch"] |
There was a problem hiding this comment.
When an operator audits or recreates the minimal Role from docs/RUNBOOK-K8S.md §4, the table—explicitly described there as the exact set of resources Warren exercises—still lists only pods, pod logs, and ConfigMaps. Omitting this new events dependency would leave the warning watcher receiving 403 responses again, so add an events row with its get, list, watch verbs and watcher rationale.
Useful? React with 👍 / 👎.
1 participant
Two small K8s-runtime fixes, found while bringing warren up on a laptop cluster (Rancher Desktop, k3s, 2 allocatable cores) to drive Azure DevOps repos.
Why we needed it
PendingwithFailedScheduling: Insufficient cpuand the run was eventually cancelled. Memory and ephemeral storage already have cluster-wideWARREN_K8S_*env defaults; CPU did not. The only override was aresourcesblock in the target repo's.warren/config.yaml, which is not always possible when the repo is not yours to change (or is a mirror).FailedSchedulingwas also invisible from warren: the pod-warning-events watcher (warren-32f8) loggedpod-event-watch disconnected; backing off before resumein a loop. The baseRolegrants pods/pods.log/configmaps but neverevents, so every list-watch got 403. Anyone deploying fromdeploy/k8s/basehas a dead watcher today.Changes
WARREN_K8S_CPU_REQUEST_MILLICORES/WARREN_K8S_CPU_LIMIT_MILLICORES(src/runtime/k8s/pod-resources.ts→resolveCpuMillicores), same precedence as the memory knob: per-projectresourcesblock > env > compiled defaults (1000/4000). Validated to the same 10m..64 CPU bounds the config schema uses; invalid values fall back, likepickMiB.pod-spec.tsconsumes it.deploy/k8s/base/rbac.yaml:get/list/watchon coreeventsfor the runs-namespace Role.docs/RUNBOOK-K8S.md: rows for the new CPU knob and for the existingWARREN_K8S_MEMORY_*knob, which was missing from the env table.pod-resources.test.tscovers env override, per-project precedence, and invalid-value fallback.Verified on the cluster above: with
WARREN_K8S_CPU_REQUEST_MILLICORES=250the pod schedules, and with the RBAC change the watcher connects andFailedScheduling/image-pull warnings land on the run's event stream.🤖 Generated with Claude Code
https://claude.ai/code/session_01Rf5tbgdUDmHzKnNXpiJLvg