feat(foreman): report missing planned children in Dispatched condition - #1743
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Defilan
left a comment
There was a problem hiding this comment.
Reviewed the diff against the issue. The implementation is faithful to the
design and the tests are real, but there is one gap a reviewer should see
before merging, and it is in the specification rather than the code.
The name-matching decision is correct and load-bearing. missingPlannedTasks
compares w.Status.Tasks against observed children by name, never by count.
That is what makes it immune to the placeholder children synthesized in
workload_escalation.go, workload_iteration.go and
workload_coder_escalation.go: those are extra observed children absent from
w.Status.Tasks, so len(children) can legitimately exceed the planned count.
A length comparison would fire on every escalation.
TestRollup_ChildrenMissing's third case ("extra child not planned") is the one
that proves this, and it is the case that would fail if anyone later
"simplified" the check into a count comparison. Worth keeping.
The gap: name matching protects against extra children, not absent ones.
markPlannedwritesw.Status.Tasks = createdsynchronously, in the same
status patch that records the Planned condition.listChildrenreads viar.List, the cache-backed client.- The escalation and iteration paths compensate for informer lag by synthesizing
placeholders. The planner path does not.
So on the first rollup after planning, a planned child whose create has not yet
reached the informer reads as missing, and the condition transiently reports
ChildrenMissing naming healthy children. In practice that means it fires
during normal startup, which is the cry-wolf behaviour the issue was filed to
prevent.
I am not asking for a change here. The report is diagnostic-only, never touches
the phase, and self-corrects on the next reconcile within seconds. The clean fix
is an uncached confirmation read on the shortfall path, and APIReader appears
zero times in this codebase today, so that is a new pattern and a larger change
than this reporting fix should carry. Tracked as #1744, including why the
timing-heuristic alternatives were rejected.
Flagging it here rather than only in the description so it is visible in the
review timeline: whoever merges this should know the condition will be noisy at
startup until #1744 lands.
Merge ordering. This PR and #1741 both edit the default: branch of the
same switch. git merge-tree reports no conflict markers, but a real git merge
fails on workload_controller.go. Merge one, rebase the other; the test file
auto-merges cleanly.
Scope is otherwise clean: classifyChildren untouched, Completed message
unchanged, phase deliberately not failed.
…ltering (defilantech#1743) The missing-planned-child report read every superseded round as gone. activeChildren drops rounds that a later attempt superseded, and those rounds never leave w.Status.Tasks: markPlanned assigns that list wholesale, appendNewTaskRefs only appends to it, and nothing trims it. rollup received the filtered slice and compared it against the untrimmed planned list, so a Workload that fix-iterated reported its base round missing, and one that coder-escalated reported its whole base attempt missing, for as long as the retry round stayed in flight. Nothing had been deleted. That is the cry-wolf the report's name matching was meant to avoid, and it fired on the two paths the design section named as the reason for matching by name. observeChildren now computes the report from the unfiltered observation and then filters, so a superseded round is seen alive. Both emissions that create children also record their refs, so a ref and its child appear together either side of the call and neither a just-planned nor a not-yet-planned task reads as missing. Tests: the superseded fix-iteration round and the escalated base attempt are not reported, and a genuinely deleted child of the live round still is, beside a superseded round that is not. All three fail when the computation is moved back after the filter. The table now drives observeChildren rather than re-deriving the caller's order. Refs defilantech#1743 Signed-off-by: Christopher Maher <chris@mahercode.io>
rollup previously classified only the children it could observe, so a child that was deleted was silently absent and the Workload stayed Dispatched forever with no indication anything was missing. Compute the planned task references recorded in w.Status.Tasks that the observed child set no longer contains and surface them on the Dispatched condition: when any planned ref is absent the Reason becomes ChildrenMissing and the message names the missing count and the missing task names. The Workload's phase is unchanged (deleting a task by hand is a legitimate operator action); this is diagnosability, not enforcement. The check matches by NAME, never by count: escalation and iteration synthesize extra placeholder children that are not in w.Status.Tasks, so len(children) can legitimately exceed the planned count and a length comparison would flap and cry wolf on every escalation. Refs defilantech#1738 Signed-off-by: Foreman Bot <chris@mahercode.io>
…ltering (defilantech#1743) The missing-planned-child report read every superseded round as gone. activeChildren drops rounds that a later attempt superseded, and those rounds never leave w.Status.Tasks: markPlanned assigns that list wholesale, appendNewTaskRefs only appends to it, and nothing trims it. rollup received the filtered slice and compared it against the untrimmed planned list, so a Workload that fix-iterated reported its base round missing, and one that coder-escalated reported its whole base attempt missing, for as long as the retry round stayed in flight. Nothing had been deleted. That is the cry-wolf the report's name matching was meant to avoid, and it fired on the two paths the design section named as the reason for matching by name. observeChildren now computes the report from the unfiltered observation and then filters, so a superseded round is seen alive. Both emissions that create children also record their refs, so a ref and its child appear together either side of the call and neither a just-planned nor a not-yet-planned task reads as missing. Tests: the superseded fix-iteration round and the escalated base attempt are not reported, and a genuinely deleted child of the live round still is, beside a superseded round that is not. All three fail when the computation is moved back after the filter. The table now drives observeChildren rather than re-deriving the caller's order. Refs defilantech#1743 Signed-off-by: Christopher Maher <chris@mahercode.io>
cb9da7a to
10a5575
Compare
What
Report planned child tasks the Workload can no longer observe, in the
Dispatchedcondition, instead of silently reporting only the survivors asordinary in-flight work.
Why
Refs #1738
rollupclassified only the children it could observe. A child that was deletedwas in no bucket at all, so the counts silently shrank and a Workload that
planned three tasks and lost one reported on two and read as healthy work in
progress. Nothing compared the observed set against what the planner actually
emitted, even though that list is already recorded in
w.Status.Tasks.Stale objects are not cosmetic on a MicroK8s/dqlite control plane: accumulated
dead objects have previously bloated dqlite and frozen kubelet's pod watch, so a
Workload that can never terminate carries real operational cost.
This is the second half of the original #1729. The first half shipped in #1736.
How
missingPlannedTasksmatchesw.Status.Tasksagainst the observed childrenby name, never by count. That distinction is the whole design:
workload_escalation.go,workload_iteration.goandworkload_coder_escalation.godeliberately synthesize extra placeholderchildren that are not in
w.Status.Tasks, solen(children)can legitimatelyexceed the planned count and a length comparison would flap on every escalation.
Name matching ignores extras and notices only a planned ref that is gone.
Name matching alone was not enough, and the second commit fixes what it missed.
The same iteration and escalation paths that synthesize extra children also
remove planned ones:
activeChildrendrops a round that a later attemptsuperseded, before
rollupsees the slice. Sincew.Status.Tasksis only everassigned wholesale by
markPlannedor appended to byappendNewTaskRefs, andnever trimmed, comparing the filtered slice against it reported the base round
of every iterated Workload, and the entire base attempt of every
coder-escalated one, as missing while the retry round ran.
observeChildrennow computes the report from the unfiltered observation and then filters, so a
superseded round is seen alive. A ref and its child are created together by
every emission, so nothing reads as missing across that call.
Tests cover the superseded fix-iteration round, the escalated base attempt, and
a genuinely deleted child of the live round beside a superseded round that is
not reported. Each fails when the computation is moved back after the filter.
When at least one planned ref is missing, the
Dispatchedcondition'sReasonbecomes
ChildrenMissingand theMessagenames the count and the missing tasknames. The Workload's phase is deliberately unchanged and it is not failed:
deleting a task by hand is a legitimate operator action, and this is
diagnosability, not enforcement.
Scoped to the
default:branch.classifyChildrenis untouched, and theCompletedcondition's message is unchanged.Known limitation, tracked as #1744
Name matching protects against extra children but not against children that are
absent because the cache has not caught up.
markPlannedwritesw.Status.Taskssynchronously whilelistChildrenreads through thecache-backed client, and unlike the escalation paths the planner path does not
synthesize placeholders. So on the first rollup after planning, a planned child
whose create has not yet reached the informer will transiently read as missing.
This is being merged with that gap open, deliberately. The report is
diagnostic-only, never changes the phase, and self-corrects on the next
reconcile within seconds. The clean fix is an uncached confirmation read before
reporting, which introduces
APIReaderto a codebase that does not currently useit anywhere, and that is a larger change than this reporting fix should carry.
#1744 covers it, including the timing-heuristic alternatives and why they were
rejected.
Tests
TestRollup_ChildrenMissingdrives the realrollupagainst a fake client:ReasonstaysChildrenInFlight, nothing reportedReasonisChildrenMissing, the message names it, andthe phase is asserted not to be
Failedw.Status.Tasks, all planned refs present:nothing reported
The third case is the anti-flap boundary that proves name matching beats
counting, and it is the one that would fail if this were reimplemented as a
length comparison.
Checklist
make testpasses locally. The clean-room gate Job ran it in full for thefirst commit (GATE-PASS); the second was verified locally, whole package
with envtest plus a sweep of
./internal/...and./pkg/foreman/...make lintpasses locally. The gate Job runs fmt, vet, lint andlint-deadcode; the second commit also passed
GOOS=linux golangci-lintgit commit -s) per DCO(DeepSeek V4 Flash, self-hosted) under band 3 of CONTRIBUTING.md, reviewed
by the automated reviewer and by the maintainer, who owns this review
conversation. The second commit was written by hand with Claude Code from
an adversarial review of the first, and verified locally with a mutation
check on each new test case.
Note for merging: #1739 has since merged and this branch is now rebased on top
of it. The conflict in the
default:branch of that switch is resolved bykeeping
conditionTypeDispatchedandsetDispatchedTerminalfrom #1739 andlayering the
ChildrenMissingreason on top of them inside the same case.