Skip to content

feat(foreman): report missing planned children in Dispatched condition - #1743

Merged
Defilan merged 2 commits into
defilantech:mainfrom
Defilan:foreman/wl-1738-child-count-v2/issue-1738
Sep 4, 2026
Merged

feat(foreman): report missing planned children in Dispatched condition#1743
Defilan merged 2 commits into
defilantech:mainfrom
Defilan:foreman/wl-1738-child-count-v2/issue-1738

Conversation

@Defilan

@Defilan Defilan commented Sep 1, 2026

Copy link
Copy Markdown
Member

What

Report planned child tasks the Workload can no longer observe, in the
Dispatched condition, instead of silently reporting only the survivors as
ordinary in-flight work.

Why

Refs #1738

rollup classified only the children it could observe. A child that was deleted
was in no bucket at all, so the counts silently shrank and a Workload that
planned three tasks and lost one reported on two and read as healthy work in
progress. Nothing compared the observed set against what the planner actually
emitted, even though that list is already recorded in w.Status.Tasks.

Stale objects are not cosmetic on a MicroK8s/dqlite control plane: accumulated
dead objects have previously bloated dqlite and frozen kubelet's pod watch, so a
Workload that can never terminate carries real operational cost.

This is the second half of the original #1729. The first half shipped in #1736.

How

missingPlannedTasks matches w.Status.Tasks against the observed children
by name, never by count. That distinction is the whole design:
workload_escalation.go, workload_iteration.go and
workload_coder_escalation.go deliberately synthesize extra placeholder
children that are not in w.Status.Tasks, so len(children) can legitimately
exceed the planned count and a length comparison would flap on every escalation.
Name matching ignores extras and notices only a planned ref that is gone.

Name matching alone was not enough, and the second commit fixes what it missed.
The same iteration and escalation paths that synthesize extra children also
remove planned ones: activeChildren drops a round that a later attempt
superseded, before rollup sees the slice. Since w.Status.Tasks is only ever
assigned wholesale by markPlanned or appended to by appendNewTaskRefs, and
never trimmed, comparing the filtered slice against it reported the base round
of every iterated Workload, and the entire base attempt of every
coder-escalated one, as missing while the retry round ran. observeChildren
now computes the report from the unfiltered observation and then filters, so a
superseded round is seen alive. A ref and its child are created together by
every emission, so nothing reads as missing across that call.

Tests cover the superseded fix-iteration round, the escalated base attempt, and
a genuinely deleted child of the live round beside a superseded round that is
not reported. Each fails when the computation is moved back after the filter.

When at least one planned ref is missing, the Dispatched condition's Reason
becomes ChildrenMissing and the Message names the count and the missing task
names. The Workload's phase is deliberately unchanged and it is not failed:
deleting a task by hand is a legitimate operator action, and this is
diagnosability, not enforcement.

Scoped to the default: branch. classifyChildren is untouched, and the
Completed condition's message is unchanged.

Known limitation, tracked as #1744

Name matching protects against extra children but not against children that are
absent because the cache has not caught up. markPlanned writes
w.Status.Tasks synchronously while listChildren reads through the
cache-backed client, and unlike the escalation paths the planner path does not
synthesize placeholders. So on the first rollup after planning, a planned child
whose create has not yet reached the informer will transiently read as missing.

This is being merged with that gap open, deliberately. The report is
diagnostic-only, never changes the phase, and self-corrects on the next
reconcile within seconds. The clean fix is an uncached confirmation read before
reporting, which introduces APIReader to a codebase that does not currently use
it anywhere, and that is a larger change than this reporting fix should carry.
#1744 covers it, including the timing-heuristic alternatives and why they were
rejected.

Tests

TestRollup_ChildrenMissing drives the real rollup against a fake client:

  • every planned ref observed: Reason stays ChildrenInFlight, nothing reported
  • a planned ref absent: Reason is ChildrenMissing, the message names it, and
    the phase is asserted not to be Failed
  • an extra observed child not in w.Status.Tasks, all planned refs present:
    nothing reported

The third case is the anti-flap boundary that proves name matching beats
counting, and it is the one that would fail if this were reimplemented as a
length comparison.

Checklist

  • Tests added/updated
  • make test passes locally. The clean-room gate Job ran it in full for the
    first commit (GATE-PASS); the second was verified locally, whole package
    with envtest plus a sweep of ./internal/... and ./pkg/foreman/...
  • make lint passes locally. The gate Job runs fmt, vet, lint and
    lint-deadcode; the second commit also passed GOOS=linux golangci-lint
  • Commit messages follow conventional commits
  • All commits are signed off (git commit -s) per DCO
  • AI assistance disclosed: authored by the Foreman coder agent
    (DeepSeek V4 Flash, self-hosted) under band 3 of CONTRIBUTING.md, reviewed
    by the automated reviewer and by the maintainer, who owns this review
    conversation. The second commit was written by hand with Claude Code from
    an adversarial review of the first, and verified locally with a mutation
    check on each new test case.
  • Documentation updated. No user-facing change

Note for merging: #1739 has since merged and this branch is now rebased on top
of it. The conflict in the default: branch of that switch is resolved by
keeping conditionTypeDispatched and setDispatchedTerminal from #1739 and
layering the ChildrenMissing reason on top of them inside the same case.

@codecov

codecov Bot commented Sep 1, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@Defilan Defilan left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the diff against the issue. The implementation is faithful to the
design and the tests are real, but there is one gap a reviewer should see
before merging, and it is in the specification rather than the code.

The name-matching decision is correct and load-bearing. missingPlannedTasks
compares w.Status.Tasks against observed children by name, never by count.
That is what makes it immune to the placeholder children synthesized in
workload_escalation.go, workload_iteration.go and
workload_coder_escalation.go: those are extra observed children absent from
w.Status.Tasks, so len(children) can legitimately exceed the planned count.
A length comparison would fire on every escalation.

TestRollup_ChildrenMissing's third case ("extra child not planned") is the one
that proves this, and it is the case that would fail if anyone later
"simplified" the check into a count comparison. Worth keeping.

The gap: name matching protects against extra children, not absent ones.

  • markPlanned writes w.Status.Tasks = created synchronously, in the same
    status patch that records the Planned condition.
  • listChildren reads via r.List, the cache-backed client.
  • The escalation and iteration paths compensate for informer lag by synthesizing
    placeholders. The planner path does not.

So on the first rollup after planning, a planned child whose create has not yet
reached the informer reads as missing, and the condition transiently reports
ChildrenMissing naming healthy children. In practice that means it fires
during normal startup, which is the cry-wolf behaviour the issue was filed to
prevent.

I am not asking for a change here. The report is diagnostic-only, never touches
the phase, and self-corrects on the next reconcile within seconds. The clean fix
is an uncached confirmation read on the shortfall path, and APIReader appears
zero times in this codebase today, so that is a new pattern and a larger change
than this reporting fix should carry. Tracked as #1744, including why the
timing-heuristic alternatives were rejected.

Flagging it here rather than only in the description so it is visible in the
review timeline: whoever merges this should know the condition will be noisy at
startup until #1744 lands.

Merge ordering. This PR and #1741 both edit the default: branch of the
same switch. git merge-tree reports no conflict markers, but a real git merge
fails on workload_controller.go. Merge one, rebase the other; the test file
auto-merges cleanly.

Scope is otherwise clean: classifyChildren untouched, Completed message
unchanged, phase deliberately not failed.

@Defilan
Defilan marked this pull request as ready for review September 1, 2026 16:00
Defilan added a commit to Defilan/LLMKube that referenced this pull request Sep 3, 2026
…ltering (defilantech#1743)

The missing-planned-child report read every superseded round as gone.

activeChildren drops rounds that a later attempt superseded, and those
rounds never leave w.Status.Tasks: markPlanned assigns that list wholesale,
appendNewTaskRefs only appends to it, and nothing trims it. rollup received
the filtered slice and compared it against the untrimmed planned list, so a
Workload that fix-iterated reported its base round missing, and one that
coder-escalated reported its whole base attempt missing, for as long as the
retry round stayed in flight. Nothing had been deleted. That is the
cry-wolf the report's name matching was meant to avoid, and it fired on the
two paths the design section named as the reason for matching by name.

observeChildren now computes the report from the unfiltered observation and
then filters, so a superseded round is seen alive. Both emissions that
create children also record their refs, so a ref and its child appear
together either side of the call and neither a just-planned nor a
not-yet-planned task reads as missing.

Tests: the superseded fix-iteration round and the escalated base attempt
are not reported, and a genuinely deleted child of the live round still is,
beside a superseded round that is not. All three fail when the computation
is moved back after the filter. The table now drives observeChildren rather
than re-deriving the caller's order.

Refs defilantech#1743

Signed-off-by: Christopher Maher <chris@mahercode.io>
rollup previously classified only the children it could observe, so a
child that was deleted was silently absent and the Workload stayed
Dispatched forever with no indication anything was missing.

Compute the planned task references recorded in w.Status.Tasks that the
observed child set no longer contains and surface them on the Dispatched
condition: when any planned ref is absent the Reason becomes
ChildrenMissing and the message names the missing count and the missing
task names. The Workload's phase is unchanged (deleting a task by hand
is a legitimate operator action); this is diagnosability, not
enforcement.

The check matches by NAME, never by count: escalation and iteration
synthesize extra placeholder children that are not in w.Status.Tasks, so
len(children) can legitimately exceed the planned count and a length
comparison would flap and cry wolf on every escalation.

Refs defilantech#1738

Signed-off-by: Foreman Bot <chris@mahercode.io>
…ltering (defilantech#1743)

The missing-planned-child report read every superseded round as gone.

activeChildren drops rounds that a later attempt superseded, and those
rounds never leave w.Status.Tasks: markPlanned assigns that list wholesale,
appendNewTaskRefs only appends to it, and nothing trims it. rollup received
the filtered slice and compared it against the untrimmed planned list, so a
Workload that fix-iterated reported its base round missing, and one that
coder-escalated reported its whole base attempt missing, for as long as the
retry round stayed in flight. Nothing had been deleted. That is the
cry-wolf the report's name matching was meant to avoid, and it fired on the
two paths the design section named as the reason for matching by name.

observeChildren now computes the report from the unfiltered observation and
then filters, so a superseded round is seen alive. Both emissions that
create children also record their refs, so a ref and its child appear
together either side of the call and neither a just-planned nor a
not-yet-planned task reads as missing.

Tests: the superseded fix-iteration round and the escalated base attempt
are not reported, and a genuinely deleted child of the live round still is,
beside a superseded round that is not. All three fail when the computation
is moved back after the filter. The table now drives observeChildren rather
than re-deriving the caller's order.

Refs defilantech#1743

Signed-off-by: Christopher Maher <chris@mahercode.io>
@Defilan
Defilan force-pushed the foreman/wl-1738-child-count-v2/issue-1738 branch from cb9da7a to 10a5575 Compare September 3, 2026 07:59
@Defilan
Defilan merged commit e9ce0ff into defilantech:main Sep 4, 2026
25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant