Skip to content

api: diagnose a stale mgmt TLS cert on host-name change (advances #5719) - #6827

Open
psaab wants to merge 22 commits into
masterfrom
fix/5719-api-hardening
Open

api: diagnose a stale mgmt TLS cert on host-name change (advances #5719)#6827
psaab wants to merge 22 commits into
masterfrom
fix/5719-api-hardening

Conversation

@psaab

@psaab psaab commented Aug 5, 2026

Copy link
Copy Markdown
Owner

Summary

Third drive on cohort #5719 (codex-review-182 C-API — diagnostic /
automation API hardening survivors). Two of the three named survivors were
already fixed on master by the earlier drives; the one real, mechanical,
in-scope gap left is the host-name half of the C001 stale-durable-cert
residual, fixed here. The third item (applied-nft truth projection) is not
fixed — scoping analysis is
a comment on #5719.
The cohort stays open and this PR says Advances, not Closes.

The PR has grown two halves since round 1, and the daemon half is now the
larger one.

The defect

The durable management cert (#1916 D6) bakes two identities into its SANs at
first generation — the HTTPS bind host and the kernel host name — and is
deliberately never re-minted (a re-mint churns remote clients' TOFU pins).
#6378 closed the silence for the bind host only.

A later set system host-name leaves the cert's DNS SAN naming the old
host. Renaming a firewall does not move its management IP, so the bind-host
check does not fire either — an operator connecting by the new name got a bare
x509: certificate is not valid for any names with nothing in the log.
Proved empirically before the fix was written (mint as old-fw / bind
10.0.0.1, reload as new-fw / same bind): PROBE reload log output = "".

This is live breakage, not hygiene. SANs are the only thing a modern client
matches on; CN-only fallback left Go's verifier in 1.15 and browsers/curl years
earlier. The handshake failed before and still fails — what changes is whether
the operator can see why, and that the remedy (rm -rf /etc/xpf/tls) appears in
the failure path instead of being undocumented.

Half 1 — pkg/api: what the LOAD path now says, and what it declines to say

generateSelfSignedCertAt's load-success path calls warnStaleLoadedCert,
which parses the leaf ONCE and warns per uncovered identity, each line naming
the identity and the cert's DNS/IP SANs — so an operator re-mints or
dismisses in one read. All checks share certCoversHost, the same strict
x509.VerifyHostname classification a remote client applies.

  • certHasNoSANs is reported first and is TERMINAL. A pair persisted by an
    older build (or placed by an operator) can carry no subjectAltName at all, and
    then it covers NOTHING — even https://localhost fails. The per-identity
    predicates gate out loopback on the premise that the durable cert always
    carries the loopback SANs, which is true only of certs this mint path
    produced; without this check the most broken certificate possible was the one
    that warned least.
  • The host-name check is gated THREE ways on the load path, not one. The
    round-1 body said "one extra condition on top of bindHostWarnable" — that is
    now wrong, and materially so, because the third gate is what makes a healthy
    box silent:
    1. hostnameSANWarnable — the name must be one a re-mint could cover
      (DNS-encodable, or an IP literal that lands in IPAddresses). A café
      host name is DROPPED from the SANs by design (isDNSSANSafeHostname, the
      api: Server.Run leaks the surviving HTTP/HTTPS listener when its sibling fails #5058 guard that stops x509.CreateCertificate hard-failing and tearing
      down the whole management server), so warning about it every reload would
      be permanent noise advising a fix that does not exist.
    2. hostName == bindHost — already reported as the bind host; repeating it is
      noise.
    3. hostNameLikelyAccessIdentity — INFERRED evidence only. A box named fw
      whose cert covers mgmt.example.com plus its management IP is verifiable
      at every URL in use; telling that operator to re-mint churns TOFU pins for
      nothing. Since this package's mint path is the only thing that puts a bare
      unqualified name in a cert, the qualification SHAPE of the DNS SANs is the
      evidence: unqualified SAN next to an unqualified kernel name means the TLS
      identity IS the kernel name and has drifted (diagnose); a qualified SAN
      next to a short kernel name means it never was (stay quiet).
      TestUnusedKernelHostNameIsSilent_6827 asserts both directions.

The rename entry point (Server.WarnStaleMgmtCertForHostName) skips gate 3
only: the operator just chose the name, which is direct evidence rather than
inference, and it is the one moment they are watching commit output. It takes
the host name as a PARAMETER rather than re-reading os.Hostname() — see the
ordering problem below.

It reads the LIVE HTTPS leg through listenerLeg.serving(), not a non-nil
pointer: an unexpected serve exit leaves the leg installed with dead set, and
a root-context shutdown leaves it installed with stopping set, and diagnosing
either would report a certificate no socket is presenting. (The LOAD path has no
leg to read — it runs inside cert generation, before one exists — so this is the
rename entry point's property, not a shared one.) Server.HTTPSServing() uses
the same predicate, so a boot HTTPS bind failure is not recorded as a converged
fingerprint, and since round 6 so does ReconcileHTTPS's same-address no-op, so
a dead leg is REBUILT rather than mistaken for a converged one.

Half 2 — pkg/daemon: reaching the diagnostic at all, and a debt that survives

The load path could never see a plain rename, for two independent reasons, and
fixing either alone is not enough:

  • Reachability — the HTTPS leg is rebuilt only when the TLS flag or the
    HTTPS bind address changes (managementReconciler.reconcileTo), so a rename
    on an unchanged endpoint reloads nothing.
  • OrderingreconcileWebManagement runs EARLY in applyConfigLocked (so
    a credential revocation survives an aborting commit) while applyHostname
    runs in the apply tail. Even a commit that DID move the HTTPS bind would have
    diagnosed the OLD kernel name.

So Daemon.applyHostname calls noteStaleMgmtCertHostName after
Sethostname succeeds, and marking-and-delivering is ONE path. What that buys,
and what each piece is for:

  • A debt, not a one-shot (staleCertPending/staleCertGen, guarded by
    staleCertMu). At BOOT the hook runs before its own dependency exists: the
    first config apply is startup phase 4 while startHTTPServer publishes
    d.mgmt much later in Run. The flag clears only when a delivery actually
    REACHED a served certificate. Clearing it whenever the delivery merely RAN
    loses the diagnosis permanently when HTTPS is off or its bind failed, because
    the next boot's applyHostname sees the name already applied and returns
    early, and the load path's inferred gate declines CROSS-shape drift by design
    (a shape-preserving rename it would still catch — but only at the next
    certificate load, a restart or an HTTPS rebind, never at the commit).
  • The name is read from the kernel at DELIVERY, never stored at rename time,
    so a deferred diagnosis is never the replay of a name captured at some earlier
    commit. That is NOT the same as "it always names the current identity", and
    round 6 corrected the body and the source where they claimed it was:
    Sethostname moves the kernel name before applyHostname records the new
    generation, so a delivery can legitimately read, validate and report a name the
    kernel has just left. Nothing keyed on the generation can close that particular
    gap.
  • The delivery is generation-fenced, before it speaks. The kernel read runs
    unlocked; the generation is sampled before it and RE-VALIDATED after it, under
    staleCertMu held across the certificate inspection and the clear. A delivery
    whose generation has been superseded abandons without warning and without
    clearing
    — it must not settle a newer rename's debt with older evidence, and
    must not emit a diagnosis naming a name a recorded rename has already replaced.
    Nothing is lost: the newer rename runs its own delivery. The window is
    reachable: the boot delivery runs on the Run goroutine OUTSIDE applySem,
    while cluster comms — started right after the mutating startup phases, before
    startHTTPServer — can drive a peer SyncApply into applyHostname.
  • Two DEFERRED retry points — the boot management start, and every
    web-management reconcile (so a later web-management https enable, or the
    rebuild of an HTTPS leg whose serve loop died, settles a debt incurred while
    nothing was serving).

Test seams: sethostname, hostnamePath, osHostname. The
already-applied early return in applyHostname is load-bearing after this PR
(noteStaleMgmtCertHostName is reachable only past it) — without it, every
commit carrying an unchanged system host-name re-fires the debt, and a box
with a genuinely stale cert WARNs on every commit.

What changed in the last round (round 9 — three of this PR's own guards were weaker than claimed)

"legDrainTimeout bounds the Shutdown" was my SECOND wrong version of that
sentence.
Shutdown's loop calls closeIdleConns() and only reaches its
ctx.Done() select if that returns false; closeIdleConns walks activeConn
under s.mu closing serially. So stalled idle TLS peers overrun the context
inside Shutdown — the same serial-close problem round 8 found in Close, one
phase earlier. It is a poll deadline and bounds neither phase in wall-clock
terms. Corrected there and swept across the eight enumerated sites plus two the
enumeration missed
, found by grepping the claim rather than working the list.

The hijack test is a TRIPWIRE, not a gate — and the counterexample is a
dependency this module already has.
golang.org/x/net is direct (go.mod:16)
and its websocket.Server hijacks inside its own handler, so
mux.Handle("/ws", websocket.Handler(h)) adds a hijacked connection with nothing
in local syntax to match. "by gate", "absence is enforced", "fails if one is
added" were all false. Narrowed everywhere claimed; the test grew a third check
(imports of packages known to hijack — x/net/websocket, net/http/httputil)
and its own doc now states what still escapes: reverse proxies, upgrade helpers,
aliases, reflection.

assertSevered treated ANY read error as closure — including its own 250 ms
read deadline.
A stream merely PAUSED, open and free to resume, would have
passed. Same fixture, wrong signal, twice: round 8 fixed a single-read version,
round 9 fixes an any-error version. It now treats a timeout as a reason to keep
waiting, requires a non-timeout error, and names which state it timed out in
(STILL STREAMING vs OPEN BUT SILENT) because those are different bugs.
Re-measured under the deleted-Close mutation: still RED on all three exits,
3649 / 1426 / 1433 reads — those streams were alive, not quiet.

Also: TryLock is a trade, not a closure (it proves the mutex is held while
the syscall runs, not that the hold is unbroken to the bump, so the split-hold
shape still passes); the starvation-mode sentence is withdrawn — normal-mode
waiters compete rather than re-queue and sync.Mutex hands ownership to a waiter
after ~1 ms of starvation, so that window is improbable to observe, not
impossible, and B2c's GREEN means "did not observe"; Shutdown stopping further
requests is qualified to HTTP/1 (h2 shutdown callbacks are async, so a stream
can open before GOAWAY).

What changed in round 8b (four corrections, two of them to round 8 itself)

Verified against the go1.26.4 source before any prose moved, because two of
these contradict what round 8 shipped an hour earlier.

Correction, 8c (8b191f154). Three of the source-comment edits below were
reported as delivered at 047c9e38d and were not in it: legDrainTimeout's
boundedness correction, drainLeg's hijack exclusion (which the gate test's
failure message points at with "see drainLeg"), drainLeg's no-Shutdown clause,
and listenerLeg.drained's "not the last act". One script held all of them and
writes once at the end; a stale anchor made it raise, so none were written, and
the build that ran next cannot see comment-only changes. pkg/api/README.md
carried the corrected prose the whole time — so the docs described corrections
the source did not have, which is precisely the divergence this section argues
is dangerous. All four are now in the source, each verified by grep rather than
by a green build.

legDrainTimeout bounds the Shutdown, not the drain. Server.Close takes
no context and closes activeConn serially (net/http/server.go:3100-3118),
and on an HTTPS leg each entry is a *tls.Conn whose ClosecloseNotify
sets its own five-second write deadline (crypto/tls/conn.go:1471-1483). A
peer stalling its receive window costs up to five seconds each, in series, so
the worst case grows with connection count. "The bound this function promises is
real" was false. The prose now says what is bounded and what is not, plus the
knock-on: Server.Wait holds lifeMu across the drain while the stale-cert
delivery waits on lifeMu under staleCertMu, so a rename racing shutdown waits
for it — and the "well inside TimeoutStopSec=20" reassurance does not hold at
scale. Bounding the sever phase for real needs per-connection tracking with
concurrent deadlined closes; not taken in an eighth round, so the claim is
narrowed instead of the code widened.

Hijacked connections escape the drain, and absence is now a GATE. Go excludes
them from both calls by documented design, and a hijacked conn leaves
activeConn, so it can outlive the drain with drained stored. Close cannot
help — the handle is gone. Two reviewers observed that pkg/api has no hijacker
and read that as closing the case; it makes it unreachable, not enforced, and
the first WebSocket endpoint inherits an invariant that has quietly stopped
holding. So: the invariant is narrowed to exclude hijacked connections, and
TestNoHijackerInThisPackage_6827 walks the AST of every production file
(text would false-positive on the comments about hijacking) for a Hijacker
assertion or a Hijack call, failing with what drainLeg would then have to
grow.

The fence cell was probabilistic while claiming it was not. Its observer
goroutine might never be scheduled inside the 100 ms window, so a pass could mean
"never looked" — and the round-7 comment asserted the opposite. Replaced with a
synchronous sync.Mutex.TryLock inside the syscall seam: no goroutine, no sleep,
the answer is the state of the mutex. 0.10s → 0.00s, and MUT-B2a still reds
it at 0.00s naming the free-mutex generation. The split-hold non-binding now
carries its mechanism instead of "a few instructions wide": the first
Unlock happens in normal mode, so the re-acquirer wins the fast-path CAS and
any woken waiter re-queues — which is why no probe of any kind lands there.

The documented lock order was inaccurate (conservative, but wrong):
warnStaleCertForHostName releases m.mu before calling into the server, so the
shape is two independent edges from staleCertMu, not a nested three-lock chain.

Smaller, all measured: drained is not the goroutine's last act (defers run
LIFO, so it precedes wg.Done) — corrected at the field and at
HTTPSLegDrainedForTest, whose answer is "exit path and drain completed", not
"goroutine returned"; d.mgmt is guarded for the stale-cert read only and the
field now says so, since the other readers are untouched; and drainLeg records
that the arm the original defect lived in called no Shutdown at all, so
correcting why Close is needed cannot be misread as undercutting why the
drain is needed.

On the fixture deadline, where the review asked for the full 5s: split on the
merits. unexpected_serve_exit — the arm the defect was in — now runs at the
production 5s so the shipped value is exercised end to end; the other two keep
the seam, because what expiry does is a property of the arm rather than of the
number; and TestLegDrainTimeoutDefault_6827 pins the shipped value so a leaked
override cannot retune production in silence. Whole cell: 5.33s.

What changed in round 8 (hostile review, MERGE-NEEDS-MINOR, no runtime defect)

F1 (blocking, and it is about THIS PR's deliverable) — the force-close was bound by nothing

Round 7 added drainLeg (bounded Shutdown, then Close on deadline), wrote that
the force-close "is not optional", and bound none of it. Measured at
d7157b7e1: deleting _ = srv.Close() left go test ./pkg/api/ -count=1
green, and reverting the retirement/root arm to the pre-round-7 bare
Shutdown left it green too. The only property under test was that the
serveErr arm called drainLeg at all.

That is not an ordinary coverage gap. The drain guarantee is what this PR
delivers, the force-close is half of it, and unbound it can be deleted by a later
edit while drained — the flag pruneRetiredLocked reads as "nothing left for a
revocation to reach" — goes on reporting true. Same lie, restored.

TestInFlightResponseIsSeveredOnEveryLegExit_6827 holds an in-flight streaming
response open across each of the three ways a leg ends and asserts it is severed.
Two fixture properties are load-bearing and are commented as such: the handler
must stream endlessly (Shutdown closes idle connections outright, so a
handler that returns leaves nothing for the force-close to do and the cell would
pass with the Close deleted), and the server must keep its absent
WriteTimeout
(adding one to make the test easier would sever the stream for a
reason unrelated to drainLeg). legDrainTimeout becomes a var — production
never assigns it — so the DEADLINE arm is reachable in 0.47s instead of 15s.

F2 — the justification named the wrong mechanism, and it argued for deleting the line

"Shutdown alone is NOT that guarantee [no connection can serve another request]"

False, and verified firsthand with a standalone probe before rewriting
anything.
Shutdown sets inShutdown, doKeepAlives() goes false, idle
connections are closed, and a connection it is still waiting on finishes its
current response and then closes. A second request on a surviving keep-alive
connection failed with unexpected EOF.

What Shutdown does not do is terminate the response already in flight:
with a 300 ms deadline it returned context deadline exceeded and the stream
kept delivering; only Close ended it. Conclusion unchanged, reason corrected at
drainLeg, at the drained field and in pkg/api/README.md — this matters
because the next reader reasons from the stated mechanism, and the round-7
version would have told them the Close was redundant.

drained is restated to what the drain actually provides: nothing this leg
accepted is still being served
— no further request AND no response in flight.
Round 7 stated only the first half, which is the half Shutdown alone already
gives.

The probe also produced a fixture fact worth keeping: after the server severs a
connection the client still returns buffered bytes (3 reads / 19 bytes before
the error), so assertSevered drains until the read errors rather than trusting
a single read.

Round-8 mutation cells

cell mutation what fired assertion or precondition? duration
F1a delete _ = srv.Close() all three subtests of the new cell assertion (assertSevered) 2.40s ×3
F1b retirement/root arm back to a bare Shutdown requested_retirement, root_context_shutdown; unexpected_serve_exit GREEN assertion 2.41s ×2
B1 (re-run) delete drainLeg from the serveErr arm held-connection cell and unexpected_serve_exit; other two subtests GREEN assertion 0.01s / 2.26s

F1b is an adjacency pair inside one test: the three subtests bind three
distinct call sites, and mutating one arm leaves the other green. The ~2.4s
durations are the assertion's own bounded wait — the property IS "severed within
the bound", so this is the case where a poll legitimately IS the assertion, not a
setup guard (see the round-7b note below).

What changed in round 7 (Codex "I would not merge this head", 2 MAJOR + 1 MINOR)

B1 (RUNTIME, blocking) — a dead leg's connections outlived a revocation

An unexpected serve-loop exit marked the leg dead and returned without any
Shutdown
. Serve closes the LISTENER on its way out, so nothing new
arrives — but the HTTP/1 keep-alive and HTTP/2 connections it had already
accepted stayed live, served by the same http.Server under the same credential
slot. drained, set by the goroutine's defer, went up anyway.

That flag is what pruneRetiredLocked spends to stop tightening a retired leg's
pinned policy, on the reading that a drained leg has nobody left for a revocation
to reach. So the round-6 recovery reconcile PINNED the dead leg's slot, the next
ReplaceAuth PRUNED it before intersecting it, and a credential the operator had
revoked went on being accepted on a socket the box believed was gone.

Provenance, measured rather than assumed. The serve-exit path never calling
Shutdown predates round 6 (it is the #6401 code), but the credential
consequence needs something to RETIRE the dead leg and pin its slot. At
ccc2f6b09 that was still reachable — ReconcileHTTPS's !want arm (a TLS
disable) and its default arm (an HTTPS-bind change) both call stopLegLocked on
a dead leg — so the shape existed, gated behind a commit that disabled TLS or
moved the bind. What round 6 added is reachability on an unchanged
configuration, through the same-address recovery and the reconciler's
!HTTPSServing() arm, which fires on every commit while the leg is dead. That is
the common case, so the fix belongs with the recovery, in this PR, rather than in
a separate pre-existing-defect ticket.

Fixed with drainLeg on ALL THREE exits (requested retirement, root-context
shutdown, unexpected exit): a bounded Shutdown, then Close when the deadline
expires. The force-close is not optional — this server runs with no
WriteTimeout by design, so Shutdown's deadline has nothing behind it: an SSE
stream or a slow reader outlives it indefinitely and Shutdown returns
ctx.Err() leaving the connection open.

Why no existing cell could see it. The three recovery cells drive the exit
with errLn, which fails from its FIRST Accept, so no connection is ever
accepted and none can survive — the case was unreachable in the fixture, not
absent from production. TestDeadLegConnectionCannotOutliveRevocation_6827 binds
a real loopback socket, holds ONE TLS connection across the kill, the recovery
and the revocation, and asserts it cannot be admitted once the leg reports
drained.

B2 (RUNTIME, blocking) — "provably current is impossible" was wrong, and it was mine

Rounds 5 and 6 argued that no generation-keyed fence could make the emitted host
name provably current, because Sethostname moves the kernel name before the
generation is recorded. That is a property of where the lock was taken, not
of the mechanism. The new Daemon.renameHostNotingStaleMgmtCert holds
staleCertMu ACROSS the syscall AND the bump, so the generation exists before
the window can open, and the two critical sections are ordered whichever way they
race:

  • delivery first — the rename cannot move the kernel name until it lets go, so
    the name being printed is still the box's;
  • rename first — the generation has already advanced, so the older delivery
    re-validates, fails, and abandons without emitting.

No lock-order inversion: the fence is a LEAF hold around the syscall seam, while
delivery takes the same mutex at the top of staleCertMu
managementReconciler.muapi.Server.lifeMu. A failed Sethostname leaves
the ledger untouched. The residual — a privileged sethostname(2) from OUTSIDE
the daemon — is now stated in place of the blanket impossibility claim, in both
READMEs and at the two functions.

B3 (MINOR) — two same-generation deliveries could both warn

The re-validation tested the generation alone. Two deliveries for ONE rename (the
boot delivery racing the rename's own attempt, or a reconcile retry racing
either) sample the same generation: the first warns and clears, the second passes
the same check and warns again over a settled debt. It now requires
staleCertPending as well.

A round-6 claim of mine that this round's measurement falsifies

The round-6 _Log.md entry claimed "every new assertion has a production line
whose deletion reds it and only it". That is false for M8. Re-measured
here over both packages, deleting leg.dead.Store(true) reds six cells:
TestUnexpectedServeExitLeavesADeadInstalledLeg_6827,
TestReconcileHTTPSReplacesADeadLeg_6827,
TestADeadHTTPSLegIsRebuiltByTheNextReconcile_6827, the new B1 cell, and the
pre-existing TestServeExitDoesNotDeadlockConcurrentWait_6401 and
TestEffectiveHTTPListenerServeExitFails. dead is a production precondition
the whole dead-leg cohort shares
, so the siblings red for a shared reason, not
an independent one. M1 has the same shape. The cells that genuinely isolate are
M4, M5a/M5b, M7 and this round's B1/B2a/B2b/B3. The M8 row below is annotated
accordingly.

Confirmed as NON-defects this round

The listener-recovery trace works end to end for NEW connections on an unchanged
configuration without a restart: an unexpected ServeTLS return closes the
listener and stores dead; there is no automatic wake, so a later
apply/reconcile is required; the daemon condition detects desired TLS plus
!HTTPSServing(); the API same-address arm also requires a serving leg; the
certificate loads, the freed address rebinds, a fresh leg installs. The precise
verdict is therefore: yes on the next reconcile with unchanged config, no
autonomous self-heal
— and, before this round, an accepted-connection lifecycle
that was still broken, which is B1.

What changed in round 6 (Codex MERGE-NEEDS-MAJOR, 7 blocking)

B1 (RUNTIME, blocking) — a dead HTTPS leg made the debt permanently undischargeable

An HTTPS serve loop that terminates unexpectedly marks its leg dead and leaves
it installed in Server.httpsLeg; it cannot be unlinked there without taking
lifeMu, which deadlocks a shutdown racing the exit (#6401 round 3). Two
independent places then read that corpse as a converged listener:

  • the reconciler's leg-changed test compared only the recorded FINGERPRINT, which
    still matched the committed endpoint, so ReconcileHTTPS was never called on
    any later commit;
  • and had it been called, its same-address arm returned nil on a non-nil
    pointer, so it would have done nothing.

HTTPS was therefore unrecoverable on an unchanged configuration for the life
of the process, and the stale-cert debt went with it: the debt clears only
against a served certificate, so it could never be discharged, and the restart
that finally rebinds HTTPS discards it. Fixed at both levels — the api-side
no-op now tests listenerLeg.serving(), and reconcileTo's HTTPS arm also fires
on next.TLS && !m.srv.HTTPSServing(), which is the boot-time question of #5561
round 14 applied to the steady state.

B2 (RUNTIME, blocking) — a delivery could log an obsolete host name

The generation was checked only on the CLEAR, after warnStaleCertForHostName
had already run. A rename landing after the kernel read therefore emitted a
diagnosis naming the previous host name, and the fence — which the round-5 body
described as making that impossible — preserved the newer debt but could not
retract the line. The re-validation, the certificate inspection and the clear now
run under one staleCertMu hold, and a superseded delivery abandons silently.
Lock order is staleCertMu -> reconciler mu -> lifeMu, and nothing under
those re-enters the Daemon. The residual, now stated rather than denied, is the
Sethostname-before-generation window described above.

B3, B4 — two guards were deletable with the suite green

d.staleCertGen++ was unbound because the race test's own osHostname stub
performed the increment: the test manufactured the state it was meant to observe
and never reached the production note path. A new subtest drives the competing
rename through noteStaleMgmtCertHostName and arranges the shape where the
increment is load-bearing — the newer rename's delivery reaches nothing (HTTPS
still down), the commit carrying it brings HTTPS up, and the OLDER delivery is
the one that finds a certificate.

The unreadable-kernel-name guard was likewise deletable: an empty name still
reaches a certificate, WarnStaleMgmtCertForHostName reports the question
ANSWERED, and the debt clears with no identity behind it while
warnStaleHostName silently declines the empty name. Three cells now cover it,
one per clause.

B5, B6, B7 — three assertions could not fail on their own fixtures

One shape: the assertion is capable of failing in principle and cannot fail on
THAT input. Fixtures replaced, not assertions.

  • failed_sethostname_is_not_diagnosed minted the cert for the name the kernel
    still has after a rejected rename, so a spurious note was silent — and cleared
    its own debt, which pending cannot see. The cert now covers neither name, and
    the LEDGER (staleCertGen, which only advances) is asserted directly.
  • TestHTTPSBindFailureIsNotReportedAsServing_6827 claimed a bare
    s.httpsLeg != nil predicate would report a failed bind as serving. It would
    not: a failed bind installs no leg under either implementation. The honest
    property (Start stays best-effort) is kept with an explicit precondition that
    the leg really is nil, and the predicate's two clauses are bound where an
    INSTALLED leg is not serving — stopping by the existing real-drain test,
    dead by a new test that drives a real serve-loop exit.
  • The no-SAN terminality assertion used a localhost / 127.0.0.1 fixture where
    both downstream identity checks decline independently, so deleting the
    terminal return changed nothing observable. It is now two subtests: the
    loopback one proves the check adds REACH where both per-identity gates decline;
    a non-loopback one makes the return observable at all.

Claim defects corrected

applyHostname does not call the diagnostic synchronously; a deferred diagnosis
is not guaranteed to name the current identity; only the rename entry point reads
a live leg (the load path runs before one exists); stopping is stored just
before Shutdown, so the socket closes an instant later and accepted requests
still drain for up to 5s; the rename call is the INITIAL attempt, not a retry
point; the load heuristic ACCEPTS the worked old-fw -> new-fw shape rather
than declining it; cluster comms start after the phase-4 apply, not inside it;
and the boot-cell helper does not stand up a "live leg that serves" — it
hand-builds state and starts no goroutine. The restart-residual cause list grows
from two causes to seven (a dead leg, an unreadable kernel name, an
API-disabled boot, a failed HTTP start and a signal-aborted startup all reach a
restart with the debt still owed).

The boot cell's limit, stated rather than implied

Codex was fair that the boot observable is weak: deleting the retry fires the
assertion, but so would replacing it with a bare osHostname() call. Two things
changed. The cell now also asserts that the management server EXISTS at the
moment of the read, which closes the "hoist it above mgmt.start" escape — that
placement can never reach a certificate. And the test says in as many words what
it does not prove: not certificate reach, not a warning, not discharge. Those
three are bound at the other retry point, where a cert-dir seam exists
(a_day2_reconcile_settles_a_debt_incurred_while_https_was_down, and now
TestADeadHTTPSLegIsRebuiltByTheNextReconcile_6827 end to end). The remaining
escape needs a pre-construction certificate seam on api.Config — a production
knob added for one test, declined.

Also

docs/refactoring-audit-current.txt regenerated: pkg/api/server.go had entered
the audit at [WATCH] (>=1500 LOC) without the heatmap being refreshed. The gate
was already RED at the round-5 head ccc2f6b09 (server.go 1606 there vs 1307 on
master; heatmap byte-identical, with no server.go row in either), measured
against a checkout at that commit. [WATCH] is advisory, so this records the
tier rather than deferring a split.

Merge with master

edefb7570 (the #6645 REST-auth wave) is merged in semantically, not textually:
both sides had independently changed listener identity, credential-publish
gating and management-state derivation since the merge base. Both sides added
Server.HTTPSServing() (git merged that CLEANLY into a compile error); one
survives, with the stronger body. Both sides fixed the absorbing boot-bind
failure; master's mechanism survives and this branch's is removed, because
on top of master's adopt-always startTo it is unreachable dead code. Full
reasoning is in the merge commit message.

Proof

Twelve mutation cells, each in a detached worktree at the pushed head with its
own GOCACHE/TMPDIR, restored and re-verified clean between cells. The whole
matrix was run twice, in two different worktrees with two different caches,
because the first pass predated a /dev/shm sweep that removed its cache and
worktree afterwards; every cell reproduced identically, and the second pass adds
a VOID verdict class (ahead of BUILD-BREAK) that fires on
no space left on device / cannot find package / input/output error so an
environment failure can never be read as a result. No cell in either pass was
VOID. The table below is the second pass. The harness
REQUIRES an applied-marker (md5 before != after) so a drifted pattern aborts the
cell instead of reporting the unmutated suite as proof, prints FULL_RC
immediately after the suite, and classifies a build break separately from an
assertion failure — one cell hit that and was re-run in three compiling forms.

cell mutation result
CONTROL none GREEN
M1 ReconcileHTTPS same-addr no-op back to s.httpsLeg != nil && … RED — TestReconcileHTTPSReplacesADeadLeg_6827 and TestADeadHTTPSLegIsRebuiltByTheNextReconcile_6827
M2 drop || (next.TLS && !m.srv.HTTPSServing()) from reconcileTo RED — TestADeadHTTPSLegIsRebuiltByTheNextReconcile_6827 (only)
M3 never abandon a superseded delivery (if false && …) RED — a_rename_landing_mid_delivery_is_not_settled, a_real_rename_mid_delivery_advances_the_generation
M3b always abandon (if true || …) — over-reach control RED — an_unraced_delivery_settles_the_debt + 8 more; the fence must narrow the clear, not suppress it
M4 delete d.staleCertGen++ RED — a_real_rename_mid_delivery_advances_the_generation and nothing else; both fence subtests stay GREEN, which is precisely the round-5 gap
M5 remove the unreadable-name guard entirely RED — all three an_unreadable_kernel_name_settles_nothing cells
M5a remove only the hostName == "" clause RED — empty_name only
M5b remove only the err != nil clause RED — error_with_a_name only
M6 applyHostname proceeds past a REJECTED Sethostname RED — failed_sethostname_is_not_diagnosed
M7 keep warnCertNoSANs but drop its terminal return RED — the_no_san_diagnostic_is_terminal; loopback_only_identities_are_still_diagnosed stays GREEN
M8 delete leg.dead.Store(true) from serveLegLocked RED — six cells, and that is the point. dead is a shared production PRECONDITION, so this cell does not isolate. Re-measured over ./pkg/api ./pkg/daemon AFTER the round-7b fixture repair, with the failing line and duration for each: serve_exit_wait_deadlock_6401_test.go:85 (0.10s), tls_stale_cert_6827_test.go:630 "HTTPSServing must report false" (0.00s), tls_stale_cert_6827_test.go:674 "the reconcile left the DEAD leg installed" (0.00s), effective_listeners_6401_test.go:154 (3.00s) — four ASSERTIONS — plus two immediate, labelled PRECONDITIONS in TestDeadLegConnectionCannotOutliveRevocation_6827 and TestADeadHTTPSLegIsRebuiltByTheNextReconcile_6827, whose premises M8 destroys outright
B1 (r7) delete drainLeg(srv) from the serve-exit arm RED — TestDeadLegConnectionCannotOutliveRevocation_6827 and nothing else in pkg/api; the three recovery cells stay GREEN because their fixture never accepts a connection
B2a (r7) leave Sethostname OUTSIDE the staleCertMu hold (the literal pre-round-7 shape) RED — TestRenameAndGenerationBumpAreOneCriticalSection_6827 and nothing else in pkg/daemon
B2b (r7) write the ledger BEFORE the syscall inside the fence RED — failed_sethostname_is_not_diagnosed only; its five sibling subtests stay GREEN
B2c (r7) hold across the syscall, RELEASE, re-take for the bump GREEN — a measured NON-binding, reported rather than glossed. The shape is still defective (name moved, generation not), but the gap is a few instructions and the probe is released at the unlock only to lose the race to the re-acquire. The guard against it is structural: one deferred unlock over a body with no intermediate release, documented at the function and in the test
B3 (r7) drop if !d.staleCertPending from the re-validation RED — a_sibling_that_already_settled_the_same_generation_silences_this_one only; its four sibling subtests stay GREEN

Adjacent cells with DIFFERENT outcomes, as anti-staleness evidence that the
harness is really re-running: M7 reds one no-SAN subtest and leaves its
sibling green, M4 reds one generation subtest and leaves the two either side
of it green, M5a/M5b each red exactly one of the three guard cells, and
round 7 adds three more — B1 (siblings green because their fixture cannot
reach the case), B2b and B3 (one subtest red, siblings green).

M8 and M1 are NOT such pairs, and round 6 was wrong to describe the matrix as
if every cell isolated. Both mutate a production precondition the whole dead-leg
cohort depends on, so their siblings red for a shared reason. Re-measured in
round 7 rather than inferred; the M8 row says what it measured.

Round 7b — the M8 reds were mostly POLLS EXPIRING, and the fixtures are fixed

Round 7's own M8 re-measurement reported six reds without reading their
durations: 5.00s / 5.01s / 5.00s / 5.01s / 3.00s and one 0.10s. A red sitting
on a round deadline value is a poll expiring, not a property failing.
Three of
those were polls on the very flag M8 deletes — startWithDeadHTTPSLeg waited 5s
for dead and fatalled, so the two pkg/api cells that SHARE it (one entry
path, not two observations) died in the fixture and neither ever reached its own
assertion; the daemon rebuild cell polled HTTPSServing(), which reads the same
flag.

Repaired at the fixtures rather than in the row. startWithDeadHTTPSLeg now
joins with Server.Wait() — deterministic, and it reads nothing under test — and
asserts only the installation. The new B1 cell does the same and then asserts
dead+drained as an immediate labelled precondition. The daemon cell cannot use
Wait (its server also has a live HTTP leg, so the join would block until
shutdown) and polls the new api.Server.HTTPSLegDrainedForTest() instead:
drained is stored by the goroutine's defer on every exit path, so it reports
that the exit happened without consulting the flag under test. Four of the six
M8 reds now witness a property; see the row.

One refinement, because the heuristic would otherwise condemn a correct cell.
TestEffectiveHTTPListenerServeExitFails reds at 3.00s and should: its poll IS
its assertion — it waits for the listener to report Failed, which is the
property. The discriminator is whether the polled predicate is the asserted
property or a precondition for it. The duration is what tells you to go and look.

The same fault was in a cell round 7 added: the fence test asserted INSIDE the
syscall seam, consuming the channel value its closing assertion then waited 5s
for, so a genuine catch reported itself twice — once truthfully and once as "the
observer never acquired staleCertMu", which is the opposite of what happened. It
now records in the seam and asserts after the rename returns, and B2a reds in
0.00s with one true message.

The earlier M5 attempt (if false {) was a BUILD-BREAK, not an assertion
result — it left err declared and unused — and is reported as such rather than
counted as a red; the three compiling forms above replace it. No other cell was
a build break: go build ./... and go vet are clean at the head, and every
red above names a failing assertion.

Prior rounds' mutations still hold and are unchanged: MUT-A/MUT-B on the two
deferred delivery call sites, the stopping drain flag, the
hostNameLikelyAccessIdentity gate, delivery-after-Sethostname ordering (the
cell survives round 7's refactor; the mutation is now hoisting
deliverStaleMgmtCertDiagnosis above the fenced rename), and startTo's
HTTPS-fingerprint clear. The one claim retired
this round is "make HTTPSServing a bare s.httpsLeg != nil" — measured GREEN,
because a failed bind installs no leg under either implementation.

Gates

gate result
go test ./... FULL_RC=0 at 987bfe918, whole tree, -count=1 (GOTMPDIR=TMPDIR=/tmp/t) — 62 packages ok, zero failures, no transient
go test ./pkg/api/ ./pkg/daemon/ -count=1 rc 0 — ok pkg/api 38.5s, ok pkg/daemon 27.5s (round-7 control)
go build ./... rc 0
go vet ./pkg/api/... ./pkg/daemon/... rc 0
gofmt -l on every touched file clean
go test ./pkg/refactoraudit/ rc 0 after regenerating the heatmap (was RED at ccc2f6b09 too)

One transient on the first ./... pass: pkg/ddns failed with
bind: address already in use on an ephemeral port — a collision with a
concurrent job on the same box, not a regression. The package passes standalone
(rc 0) and in the re-run above.

No Rust touched, so the cargo leg is not implicated.

Docs

pkg/api/README.md and pkg/daemon/README.md describe both identities, all
three load-path gates, the debt ledger's properties, and which observable binds
each of the two deferred retry points and why they differ. Round 6 corrected
every claim listed above in BOTH files as well as in the source, added the
dead-leg rebuild to the daemon README's reconcile-discipline section, widened
the restart-residual cause list, and stated the boot cell's limit where a reader
will meet it. Round 7 adds the leg-drain bullet to pkg/api/README.md, replaces
the "no fence can close that window" paragraph in BOTH READMEs with what the
fence now proves and the one residual it does not, and records the duplicate-warn
arm. _Log.md updated, including an explicit correction of the round-6
mutation-isolation claim.

Not in scope

The applied-nft truth projection item is left open with a full analysis on the
issue: hostInboundEnforced is process-local with no exported accessor, and
ReadHostInboundDenyCounters returns (nil, nil) for BOTH a counter-less
fail-closed fence and an absent table — so REST publishes an authoritative
host_inbound_kernel_denies: 0 while a fence is actively dropping. Fixing it
means choosing which kernel states alarm, on which channel, and whether to infer
from the kernel or wait for the daemon latch. That is a design call, not a
mechanical guard.

Advances #5719.

🤖 Generated with Claude Code

Paul Saab and others added 5 commits August 5, 2026 08:21
The durable self-signed management cert (#1916 D6) bakes TWO identities
into its SANs at first generation — the HTTPS listener bind host and the
kernel host name — and is deliberately never re-minted, because a
re-mint would churn remote clients' TOFU pins. #6378 closed the silence
for the first identity: on the load-success path generateSelfSignedCertAt
warns when the current bind host is not covered by the loaded leaf.

The second identity was still unchecked. A later `set system host-name`
leaves the cert's DNS SAN naming the OLD host, and because renaming a
firewall does not move its management IP, the bind-host check does not
fire either — so an operator connecting by the new host name got a bare
"x509: certificate is not valid for any names" with NOTHING in the log.
Verified empirically before writing the fix: mint as `old-fw` with bind
`10.0.0.1`, reload as `new-fw` with the same bind, captured log output
was the empty string.

The load-success path now calls warnStaleLoadedCert, which parses the
leaf ONCE and emits a diagnostic per uncovered identity, each naming the
identity and the cert's DNS/IP SANs so the operator can re-mint (remove
/etc/xpf/tls). Both checks share certCoversHost, i.e. the same strict
x509.VerifyHostname classification a remote client applies.

The new check is gated by hostnameSANWarnable, which layers one extra
condition on bindHostWarnable: the host name must be one a re-mint could
actually cover — DNS-encodable, or an IP literal (which lands in
IPAddresses). A non-ASCII host name such as `café` is DROPPED from the
SANs by design (isDNSSANSafeHostname, the #5058 guard that keeps
x509.CreateCertificate from hard-failing and tearing down the whole
management server), so it is uncoverable by any re-mint; warning about it
on every reload would be permanent noise advising a fix that does not
exist.

Re-minting itself remains deferred — it needs a mint-ordering /
invalidation hook plus a decision on churning the durable TOFU pin. This
change closes only the diagnostic half.

Validation: two mutations, each preceded by `go build` + `go vet` rc=0 so
the RED is an assertion and not a compile break. Deleting the host-name
warn branch fails host_name_change_warns,
ip_literal_host_name_change_warns and bind_host_warning_still_fires while
the pre-existing TestLoadedCertBindHostMismatchWarns keeps PASSING (the
mutation is scoped to the new guard). Dropping the encodability gate from
hostnameSANWarnable fails unencodable_host_name_is_silent plus three
TestHostnameSANWarnable rows, proving that gate is not vacuous. Each file
was restored and touched, and both went GREEN again. `go test
./pkg/api/... ./pkg/grpcapi/...` and the full `go test ./...` exit 0;
gofmt clean. pkg/api/README.md previously claimed this diagnostic already
covered the host-name path; it now describes both identities and their
distinct gates truthfully.

Advances #5719.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015oARShYtiJJ2H4UB4nXGqi
The #5719 C001 diagnostic was written for `set system host-name`, and
that is the one case it could not observe. Three defects, all found by
review of PR #6827:

REACH + ORDERING. warnStaleLoadedCert is called only while a certificate
is being loaded, and the HTTPS leg is rebuilt only when the TLS flag or
the HTTPS bind address changes (managementReconciler.reconcileTo). A
plain host-name commit on an unchanged bind reloads nothing, so the
appliance stayed silent until a restart or a later HTTPS rebind. And
because reconcileWebManagement runs EARLY in applyConfigLocked (before
the dataplane apply, so a credential revocation lands even on an
aborting commit) while the kernel name is set in the apply tail, even a
commit that DID move the HTTPS bind would have diagnosed the OLD name.
Server.WarnStaleMgmtCertForHostName is a second entry point that reads
the LIVE HTTPS leg's served certificate — never the httpsServer
construction template, which survives a TLS disable — and diagnoses an
EXPLICIT host name. Daemon.applyHostname calls it inline with a
successful Sethostname, passing cfg.System.HostName; taking the name as
a parameter removes the ordering hazard by construction rather than by
placement. syscall.Sethostname and /etc/hostname became package seams so
the wiring is unit-testable without CAP_SYS_ADMIN.

NO-SAN CERTIFICATE. A CN-only pair — persisted by an older build, or
placed by an operator — matched neither warnable predicate, because both
gate out loopback on the premise that the durable cert always carries
the loopback SANs. True of certs this mint path produced; not true of
whatever is on disk. So the most broken certificate possible produced
total silence, while modern clients reject it for every URL including
https://localhost. certHasNoSANs reports it first and terminally: the
per-identity lines would each report "does not cover X" for a cert that
covers no X at all.

FALSE POSITIVE. The kernel host name was diagnosed unconditionally, so a
box named `fw` whose cert covers mgmt.example.com plus its management IP
— verifiable at every URL in use — was told to re-mint, churning remote
clients' TOFU pins to fix nothing, and a diagnostic that fires on a
healthy box gets muted. hostNameLikelyAccessIdentity narrows the LOAD
path by naming shape: this package's mint path is the only thing that
puts a bare unqualified name in a certificate, and what it puts there is
the kernel host name, so an unqualified SAN next to an unqualified
kernel name means the TLS identity IS that name and has drifted, while a
domain-qualified SAN next to a short kernel name means the TLS identity
is independent of it. The RENAME entry point deliberately skips the
heuristic: the operator just chose the name, which is evidence rather
than inference. The remaining load-path noise is bounded to one line per
HTTPS bind, and the message now names the identities the certificate
does cover so it can be dismissed in one read.

pkg/api/README.md claimed the kernel host name was diagnosed after
`set system host-name` and that tests covered the host-name-change
paths, and the _Log entry claimed the diagnostic half was closed. None
of that was true; the docs now describe the two entry points and the
narrowing rule, and the earlier log entry carries an explicit
correction.

Validation: five mutations, each with `go build ./...` + `go vet` rc=0
first so every RED is an assertion, not a compile break, and each file
restored from a byte snapshot verified with sha256sum -c. Deleting the
applyHostname call reds renamed_box_is_diagnosed with an empty log;
passing the pre-rename name reds the "must observe the NEW kernel host
name" assertion; neutralizing certHasNoSANs reds both no-SAN subtests;
deleting the narrowing gate reds unused_qualified_cert_identity_is_silent
while its matched positive control drifted_short_name_warns keeps
passing; downgrading the rename evidence to hostNameInferred reds
operator_chosen_name_overrides_the_heuristic. `TMPDIR=/tmp go test
./pkg/api/... ./pkg/daemon/...` and `go test ./pkg/refactoraudit/` exit
0; gofmt clean on every touched file.

Advances #5719.
The hostNameLikelyAccessIdentity narrowing has a second edge that the
previous commit stated only in review, not in the shipping artifact.

A rename that CROSSES the qualified/unqualified boundary (`old-fw` ->
`newfw.example.com`) is diagnosed at the commit, because the rename entry
point skips the heuristic, but never on a later boot: from then on the
load path sees a shape mismatch and stays quiet. For a box that was
already in that state before this diagnostic shipped there was no commit
to catch it, so it is never diagnosed at all.

The gap is bounded on two sides — it needs drift that pre-dates the
feature AND a rename that crossed the boundary. Shape-preserving drift,
unqualified to unqualified, including the worked `old-fw` -> `new-fw`
case, is still caught on every boot.

Closing it would take a one-shot sweep at the first boot after an
upgrade, which means upgrade-scoped persistent state for one narrow
class, and it would fire on exactly the boxes where the heuristic cannot
tell whether the name is in use — reintroducing the false positive at the
least convenient moment. Weighed and declined; the reasoning belongs on
the function rather than in a review thread, where the next reader can
see the choice was made deliberately rather than missed.

TestUnusedKernelHostNameIsSilent_6827/unused_qualified_cert_identity_is_
silent is the executable statement of the residual, since the load path
cannot distinguish the pre-existing drifted box from the healthy
configuration that subtest asserts. Its doc comment now says so, so
closing the gap later is a deliberate edit to a named subtest rather
than a surprise RED.

Comments and prose only: the diff on pkg/api/server.go is entirely
within a doc comment, no production statement changed.

Validation: `TMPDIR=/tmp go test ./pkg/api/... ./pkg/daemon/...` and
`go test ./pkg/refactoraudit/` both exit 0 from a real exit code;
pkg/api/server.go 1390 -> 1408, below the 1500 audit-entry threshold, so
no tier change and no heatmap regeneration; gofmt clean.

Advances #5719.
Two reach findings from the gate at 78b70ed. Neither touches the
narrowing heuristic, which was reviewed and passed; both are about
whether the diagnostic has a live certificate in front of it at all.

THE HOOK WAS ITSELF ORDERED BEFORE ITS DEPENDENCY. The first config
apply is startup phase 4 — setupDataplaneAndInitialConfig ->
applyConfig -> applyTailReconciles -> applyHostname — while
startHTTPServer constructs d.mgmt much later in Run. A `system
host-name` applied at boot therefore reached a nil reconciler. Skipping
on nil would have reproduced the exact silence this diagnostic was
added to remove, because the surviving fallback is the load path's
inferred heuristic, and that declines precisely this shape: a cert
naming oldfw.example.com next to a new kernel name new-fw is silent
there by design. So the name is parked on the Daemon under a mutex and
delivered by drainDeferredStaleCertHostName immediately after the boot
management start, still carrying operator-set evidence.

The drain consumes the parked name whether or not HTTPS came up. That
is deliberate rather than lossy: the diagnostic reports on a
certificate the box is serving, so with no leg there is nothing to be
stale, and holding the name would let some later HTTPS enable replay a
rename from arbitrarily far in the past as though it had just
happened.

A TERMINATED LEG WAS STILL TREATED AS LIVE. An unexpected serve exit
sets only leg.dead and leaves the leg installed in s.httpsLeg —
marking it under lifeMu would deadlock a shutdown racing the exit — and
a leg retiring under a requested shutdown is likewise installed while
it drains, with dead never set on that path. The previous predicate was
a non-nil pointer test, so both states warned about a certificate no
socket is presenting: the same false positive the httpsServer
construction template was rejected for.

listenerLeg.serving() now answers that question properly — non-nil,
holding a listener, serve loop not exited, no retirement requested. It
is deliberately stricter than EffectiveHTTPAddr's inline check, which
omits the stopCh test because `show system services` should still
report the address a draining leg is finishing on. The two questions
differ, so they are not folded into one predicate.

Adopting serving() turned every positive API subtest silent until the
test helper was fixed to build a genuinely serving leg rather than a
bare struct — the predicate binding its own fixtures.

Validation: two mutations, each with `go build ./...` + `go vet` rc=0
first so the RED is an assertion, each file restored from a byte
snapshot verified with sha256sum -c. Replacing the park with a bare
nil-check reds boot_rename_is_diagnosed_once_mgmt_is_up with an empty
log — the original silence verbatim — plus last_name_wins_while_mgmt_
is_down. Reverting serving() to the pointer test reds all three of
dead_leg_is_not_diagnosed, draining_leg_is_not_diagnosed and
leg_without_a_listener_is_not_diagnosed, each logging a full host-name
warning for a leg serving nothing. `TMPDIR=/tmp go test ./pkg/api/...
./pkg/daemon/...` and `go test ./pkg/refactoraudit/` exit 0; gofmt
clean.

Advances #5719.
Three findings from the gate at 608e570. All three are the same
mistake in different places: treating "the code ran" as "the question
was answered".

THE DEBT WAS CLEARED ON DELIVERY, NOT ON SUCCESS. The previous round
consumed the parked host name whenever the drain executed, on the
reasoning that with no serving leg there is nothing to be stale and the
load path would catch it later. The second half is false, and this PR
had already written down why. If the HTTP start failed, m.srv was never
published and reconcileTo returns early on a nil server, so no later
commit recovers it. If HTTPS is off or its bind failed, there is no
leg to diagnose against. In both cases the next boot's applyHostname
sees the name already applied and returns early, and the load path's
inferred heuristic declines cross-shape drift by design — the residual
documented on hostNameLikelyAccessIdentity. So the diagnosis was lost
permanently, not deferred.

The certificate is durable on disk. Staleness outlives the listener,
so the debt has to outlive it too. WarnStaleMgmtCertForHostName now
reports whether it reached a served certificate, and the pending flag
clears only on that. Delivery is retried at the rename, at the boot
management start, and on every web-management reconcile, so enabling
HTTPS months later still settles an outstanding diagnosis.

THE MARK AND THE DELIVERY WERE NOT ATOMIC. The old shape branched on a
nil d.mgmt outside staleCertMu while startHTTPServer published d.mgmt
and drained separately, so a rename landing in that window neither
refreshed the parked state nor got diagnosed. Rather than widen the
lock, the stored name is gone: the kernel host name is read at delivery
time. A deferred diagnosis now always describes the identity the box
has when it speaks, which also removes the replay hazard that motivated
consuming the name in the first place. Marking and attempting are one
path, so there is no window left to drop a rename in.

serving() MISSED THE STATE THAT OCCURS AND COVERED ONE THAT CANNOT.
The root-context arm of serveLegLocked drains and returns having set
neither dead nor stopCh, leaving the leg installed with every flag
clear — that is the state a live diagnostic actually meets after
shutdown. Meanwhile a requested retirement is unobservable through
s.httpsLeg by construction: the disable arm clears the field before
retiring, and the rebind arm installs the replacement first. So the
stopCh arm guarded a state that cannot be reached while the reachable
one slipped past. It is replaced by an exited flag stored from a defer
over the whole serve goroutine, which no exit path can skip. The
enumeration was previously taken from the paths that set flags rather
than the paths that return without setting any.

Validation: two mutations, each with `go build ./...` + `go vet` rc=0
first so the RED is an assertion, each file restored from a byte
snapshot verified with sha256sum -c. Clearing the debt whenever
delivery runs reds debt_survives_a_delivery_that_reached_nothing;
dropping exited from serving() reds root_shutdown_exited_leg_is_not_
diagnosed and reports_whether_it_reached_a_certificate. `TMPDIR=/tmp
go test ./pkg/api/... ./pkg/daemon/...` exit 0; gofmt clean.

Advances #5719.
Paul Saab added 2 commits August 5, 2026 12:32
…tion-safe

Round 5 of the #6827 gate. Two absorbing start paths, a debt clear that
could settle a newer rename with an older delivery, a predicate that
still counted a draining leg as serving, and three production edits that
no test bound.

A BOOT HTTP BIND FAILURE WAS ABSORBING. startTo assigns m.srv only after
Start succeeds, so every later reconcile returned at the m.srv == nil
early exit — clearing the cause never recovered the management plane.
startTo now retains the root context even when the start fails, and
reconcileTo retries the construction instead of treating nil as
terminal.

AN HTTPS BIND FAILURE REPORTED SUCCESS. api.Server.Start logs the
failure and returns nil, because HTTPS is best-effort at boot and the
HTTP plane should stay up. startTo recorded the HTTPS fingerprint on
that nil anyway, so an identical later reconcile saw no change and never
retried the bind. Server.HTTPSServing is the honest signal, and
startLocked leaves the fingerprint unrecorded when the leg did not come
up — the same retry-debt posture reconcileTo already uses. management.go
is the only caller of Start, so no other site was reasoning about a
listener that does not exist.

THE DEBT CLEAR WAS NOT GENERATION-SAFE. Delivery read the flag under the
mutex, ran the hostname read and certificate inspection unlocked, then
cleared unconditionally — so a rename committed in that window was
settled by the older delivery and lost. A generation counter is claimed
at read and re-checked before the clear. d.mgmt is now published under
staleCertMu, the mutex the delivery path already reads it through, so
that read is memory-model safe rather than a benign-looking race.

A DRAINING LEG COUNTED AS SERVING. Shutdown closes the listener and then
waits up to five seconds for active requests. No new client can reach
the certificate in that window and the process is going down, so a
staleness warning there is noise. Drain now counts as not-serving, set
at the top of both drain arms, and the decision is written where the
predicate is defined.

The exited flag added last round is deleted. With the drain flag
covering both retirement arms and dead covering self-termination, every
exit path is already covered, so exited became unbindable — deleting it
left the whole suite green. Keeping an arm no test can drive is exactly
what made the previous round's predicate look complete, so it goes.

The gate was right that three edits were unbound. Two are now bound by
tests that drive the production transition rather than storing the state
themselves: the drain flag by starting a real leg, cancelling the root
context and joining the goroutine, and the rename ordering by observing
the kernel name at the moment the note fires. The third — the
per-reconcile retry call site — is still unbound and is reported as
such, with the two failed approaches recorded in the test file rather
than a vacuous passing test left in its place.

Advances #5719.
@psaab

psaab commented Aug 6, 2026

Copy link
Copy Markdown
Owner Author

Hostile gate at 88d113e6b — DO-NOT-MERGE until F1/F2 are bound. The diagnostic itself is correct.

Traced every hop from set system host-name through applyHostname (daemon_system.go:432, inside the success path, after the kernel name moved) -> noteStaleMgmtCertHostName -> deliverStaleMgmtCertDiagnosis (which re-reads the kernel at delivery, so it cannot report a stale name) -> WarnStaleMgmtCertForHostName -> slog.Warn. The round-1 "reads a field nothing writes" defect has not returned in another spelling: severing the hook reds.

It also fires only when it should. certCoversHost calls x509.Certificate.VerifyHostname, which consults only DNSNames/IPAddresses — Go dropped the CN fallback in 1.15 — and routes patterns through matchHostnames, so a *.example.com cert stays silent on a rename to new-fw.example.com. No false positive on correctly-configured wildcard deployments. IPv6 brackets are stripped before the bind-host suppression compares, IP SANs are routed past the DNS-safety predicate, and a name the mint path would drop as non-encodable stays silent — symmetric with the mint side. Both prescribed directions (always-stale, never-stale) red with assertions.

Nothing re-mints, refuses a commit, drops a connection, or changes TLS parameters; the #1916 D6 no-auto-regenerate contract is intact.

Three findings:

  • F1 (major, test) — the generation fence at management.go:341-345 is unbound, and its test is a tautology: TestDebtClearIsGenerationSafe_6827 re-implements the comparison in its own body and asserts on its own arithmetic (2 == 1 is false, so the assertion holds unconditionally). The production comparison is never called. Mutating the guard to an unconditional clear leaves both packages green; the opposite direction reds. And the fence guards a real race — the boot delivery at daemon_run_servers.go:493 runs on the Run goroutine outside applySem, while cluster comms start earlier at daemon_apply_tail.go:254, so a peer SyncApply can bump the generation inside the unlocked osHostname() window.
  • F2 (major, test) — two runtime behaviour changes ride under the "diagnose" title with zero coverage: reconcileTo now retries construction when m.srv == nil (a boot HTTP bind failure was absorbing, now recoverable), and startLocked clears the HTTPS fingerprint on a failed bind so later reconciles issue ReconcileHTTPS. Both can be reverted wholesale with the suite green. TestHTTPSBindFailureIsNotReportedAsServing_6827 binds the predicate but calls it directly — the reconciler's use of it is unbound. Both changes are correct and not fail-open (the retry builds from desired(cfg) with the web-management: the REST/config API binds to a non-loopback address without requiring api-auth → unauthenticated mutating REST + /metrics from the network (fable-161 F-155) #4047/api: --api-addr non-loopback bind bypasses the #4047 no-auth loopback clamp when no web-management config block exists #5127 loopback clamp and retains no prior listener); they just need binding.
  • F3 (text)pkg/api/README.md, listener.go:41/51 and four sites in the test file describe an exited field stored "from a defer over the whole serve goroutine". No such field exists; the predicate tests dead and stopping, and stopping is stored explicitly at listener.go:130 at the start of the drain. The behaviour is right — all three goroutine exits are covered — but one of those sites is a "RED on revert" recipe naming code that isn't there.

Credit where due: the PR already discloses one unbound call site in a comment at hostname_stale_cert_6827_test.go:315-329, with the failed attempts written out. That disclosure is accurate — it just doesn't cover F1 or F2.

Paul Saab added 5 commits August 6, 2026 01:50
This change carries no runtime edit. It closes three gate findings on the
#6827 range, two of which are load-bearing test gaps: a guard whose only
test never called it, and two runtime behaviour changes nothing bound at
all.

F1 — the debt-clear generation fence was UNTESTED. Its only test,
TestDebtClearIsGenerationSafe_6827, re-implemented `d.staleCertGen == gen`
in its own body and asserted on its own arithmetic (it set the generation
to 2, then evaluated `2 == 1` itself), so the production comparison was
never reached. Measured: mutating the guard to
`if true || d.staleCertGen == gen` — an unconditional clear — left
pkg/api and pkg/daemon green, exit 0.

The fence guards a reachable race, not dead code. applySem serializes
commits with each other, but the boot delivery in daemon_run_servers.go
runs on the Run goroutine OUTSIDE applySem, and cluster comms start
earlier (inside the phase-4 boot apply, which precedes startHTTPServer),
so a peer SyncApply -> applyHostname can bump the generation while the
boot delivery sits in its unlocked osHostname() + cert-inspection window.
The test now drives deliverStaleMgmtCertDiagnosis for real and lands the
rename from INSIDE that window, using osHostname as the seam because
management.go reads it at exactly that point. A second subtest is the
negative control: with no concurrent rename the debt MUST settle, so the
pair distinguishes the fence from an unconditional clear on one side and
a never-clear on the other.

F2 — this range also changes management-listener RECOVERY, which the
"diagnose" framing understates:

  - reconcileTo used to `return nil` whenever m.srv was nil, so a boot
    HTTP bind failure was ABSORBING: clearing the cause never brought
    management back without a daemon restart. It now retries construction
    via startLocked.
  - startLocked now unrecords cur.tls/cur.httpsAddr when the boot HTTPS
    bind failed. api.Server.Start returns SUCCESS in that case (HTTPS is
    best-effort at boot), so recording the fingerprint wholesale pinned
    the reconciler to a listener that does not exist and an IDENTICAL
    later reconcile saw "no change" and never issued ReconcileHTTPS.

Both were revertable wholesale with the suite green.
TestHTTPSBindFailureIsNotReportedAsServing_6827 binds the HTTPSServing
predicate but calls it DIRECTLY — the reconciler's USE of it was unbound,
and the srv==nil retry was unbound entirely. The new
TestMgmtListenerRecoversFromAFailedStart_6827 binds both at the call site
with a failing listener factory, and carries an over-reach guard on each
side: a disabled API (no root context, or an empty HTTP bind) must NOT be
constructed by a reconcile, and a SUCCESSFUL boot HTTPS bind must still
record its fingerprint so an identical reconcile stays a no-op. The
behaviour itself is unchanged and is not fail-open — the retry builds
from desired(cfg) with the #4047/#5127 loopback clamp intact and retains
no prior listener, so it cannot produce a non-loopback no-auth listener.

F3 — pkg/api/README.md and three listener.go comments documented a field
named `exited`, "stored from a defer over the whole serve goroutine".
There is no such field. The shipped predicate tests `dead` and
`stopping`, and `stopping` is stored EXPLICITLY at the top of the drain
arm, before Shutdown. Corrected there and at four sites in
tls_stale_cert_6827_test.go, one of which gave a "RED on revert: delete
the defer in serveLegLocked" recipe for code that is not present; the
replacement recipe was verified firsthand. Two adjacent clauses claiming
a root-context shutdown "sets no flag at all" were corrected too — it
sets `stopping`.

Docs: pkg/daemon/README.md had no coverage of this work and was untouched
by the range even though pkg/daemon/management.go gained ~120 lines. It
now documents the boot-start recovery (both halves) and the
stale-certificate diagnosis — the debt, the delivery-time kernel read,
and the generation fence.

Validation: go build ./... 0; go test ./pkg/api ./pkg/daemon ./pkg/config
-count=1 0; go test -race ./pkg/api ./pkg/daemon -count=1 0; go vet
./pkg/api ./pkg/daemon 0; go test ./pkg/refactoraudit/... 0. Every
mutation red below is an ASSERTION failure, not a build break, and
management.go was restored byte-identical after each (git diff empty):

  if true || d.staleCertGen == gen                -> F1 subtest 1 FAILS,
                                                     subtest 2 PASSES
  if false && d.staleCertGen == gen               -> subtest 1 PASSES,
                                                     subtest 2 FAILS
  if true || m.rootCtx == nil || next.Addr == ""  -> HTTP-recovery subtest
                                                     FAILS, other 3 PASS
  if false && next.TLS && !srv.HTTPSServing()     -> HTTPS-retry subtest
                                                     FAILS, other 3 PASS
  delete leg.stopping.Store(true)                 -> the F3 recipe's test
                                                     FAILS as documented

Advances #6827.
The re-gate returned MERGE-READY with two minors that share one root: the
`osHostname` seam was introduced twelve lines above a read that still called
`os.Hostname()` directly.

The already-applied early return had zero coverage and is load-bearing.
`applyHostname` returns early when the kernel name already equals the
configured one. That guard was innocuous before this PR; it is now
load-bearing, because `noteStaleMgmtCertHostName` is reachable only past it.
Deleting it left the whole pkg/daemon and pkg/api suite green — while in
production every commit carrying an unchanged `system host-name` would
re-fire the debt and its delivery, so a box with a genuinely stale durable
certificate would emit the "does not cover the current host-name" warning on
EVERY commit. Muting a real diagnostic by repeating it is the failure mode
`hostNameLikelyAccessIdentity`'s own design avoids on the load path, and
three doc blocks added by this PR already assert the guard as fact.

It was untestable because the seam was half-applied. The guard read
`os.Hostname()` while the rest of the function read `osHostname`, so no test
could present a kernel name differing from the configured one. Completing
the seam is a production no-op — same function, one indirection — and makes
the guard reachable from a fixture.

That half-applied seam also let a fixture describe an impossible state.
`boot_rename_is_diagnosed_once_mgmt_is_up` stubbed the kernel as already
reporting `new-fw-6827` and then asked `applyHostname` to rename TO it. In
production the early return fires first and `sethostname` is never called,
so the sequence cannot occur; it only ran because the guard bypassed the
stub. The fixture now uses the stateful-stub pattern already established in
this file by `TestRenameIsNotedOnlyAfterSethostname_6827`: the stub reports
the PRE-rename name and the fake `sethostname` advances it.

Validation. go build ./... and go test ./pkg/daemon ./pkg/api ./pkg/config
-count=1 both exit 0; gofmt clean. Two mutations, both reddening the new
subtest as ASSERTIONS at hostname_stale_cert_6827_test.go:200:

  Neutralising the early return: "a commit whose host-name already matches
  the kernel must not call sethostname; got [steady-fw-6827]".

  Reverting the seam to os.Hostname(): the same red — which is the point.
  The half-applied seam is precisely what made the guard untestable, so the
  new subtest binds the seam and the guard together.

The fixture uses a certificate that does NOT cover the steady name, so the
"no warning" half of the assertion is not true for free.

Advances #6827.
Resolve the sole conflict, _Log.md, by union: every entry from both
sides is retained and none is rewritten. The file is no longer
append-ordered, so line counts and prefix checks say nothing useful
about the result; the resolution was verified structurally instead, by
confirming that each pre-merge side diffs into the merged file with
add-hunks only and no changed-or-deleted hunk on either side.

Every other path merged without conflict. Because a clean textual
auto-merge can still break compilation when a signature moves on one
side, that was confirmed by building rather than by inspection: go build
./... clean on the merged tree.

Advances #6827.
psaab pushed a commit that referenced this pull request Aug 13, 2026
Master's #6645 REST-auth wave (edefb75) and this branch both changed
listener identity, credential-publish gating and management-state
derivation since the 4960e7b merge base, so this is resolved by
semantics rather than by taking whichever side looked newer.

DUPLICATE METHOD GIT MERGED CLEANLY

Both sides independently added Server.HTTPSServing(), at different
offsets in listener.go, with the same justification: Start returns a nil
error when only the HTTPS bind fails, so a caller that records a
converged HTTPS fingerprint on that basis pins itself to a listener that
does not exist and never retries. The textual merge produced two
definitions and a compile error. One survives, with the branch's
strictly stronger body -- listenerLeg.serving(), which adds `ln != nil`
and `!stopping` to master's `!= nil && !dead` -- and a doc comment
carrying both rounds' reasoning. Weakening it back to master's predicate
reds four assertions in pkg/api, so the stronger reading is bound rather
than merely written down.

ONE RETRY MECHANISM SURVIVES, NOT TWO

#5561 round 14 (master) and #6827 round 5 (this branch) fixed the same
defect -- a boot bind failure was absorbing, so clearing the cause never
brought management back without a daemon restart -- by different
mechanisms. Master's survives: startTo ADOPTS the api.Server whether or
not the bind succeeded, so the retry runs through reconcileTo's ordinary
path, which asks where the live legs are before it publishes a
credential grant.

The branch's mechanism -- retain the root context and re-CONSTRUCT from
reconcileTo's `m.srv == nil` branch -- is removed rather than kept
alongside it. It would have built a fresh api.Server with the committed
Auth already installed, bypassing everyLiveLegNamedBy and
publishNilDirectionLocked entirely; and master's b27ab99 (an absent
HTTP leg is not a listener at an unnamed address) is reachable only
BECAUSE the server is adopted, so reinstating the nil-srv early return
would have re-stranded exactly the state that commit fixed. Two fences
that can disagree are worse than either alone.

m.rootCtx and startLocked are gone; the merged startTo and reconcileTo
are byte-identical to master's.

A PROSE CLAIM THE MERGE FALSIFIED

serving()'s comment said "there is no third defer-set flag: it would be
unbindable". Master added exactly that -- drained, stored from a defer so
a retired leg can be reaped from s.retiring without joining it. The
comment now explains why drained is the wrong answer to serving()'s
question rather than denying it exists: it is the goroutine's LAST act,
so it reads false throughout the 5s drain, which is the whole window in
which the socket is closed but the pointer is live.

DOCS AND TESTS

pkg/daemon/README.md loses the branch's "Failed BOOT start" bullet, which
described the removed mechanism; master's "Boot retry debt is real debt"
bullet documents the surviving one. The #6827 stale-cert-on-rename
section is unchanged.

management_recovery_6827_test.go is reduced to the two over-reach guards
that are still true and that management_bootretry_5561_test.go does not
cover: a reconcile arriving before any start must not construct a server,
and a SUCCESSFUL boot HTTPS bind must still record its fingerprint. The
two subtests that asserted m.srv stays nil after a failed boot bind are
deleted -- they pinned the mechanism that lost.

VALIDATION

go build ./... rc 0; go vet ./pkg/api/... ./pkg/daemon/... rc 0;
go test ./pkg/api/ ./pkg/daemon/ rc 0 with TMPDIR=/dev/shm/t. Two
mutations driven to RED on real assertions, build and vet clean under
each, then restored and re-run GREEN: widening the HTTPS-fingerprint
clear to `if next.TLS` reds three tests including the retained guard, and
weakening serving() reds four assertions as above.

The _Log.md union was verified structurally BEFORE this commit's own
entry was prepended: 1595 `^## ` headers = 1581 (branch) + 1587 (master)
- 1573 (merge base), with the header multiset equal to branch + master -
base exactly.

Advances #5719.
@psaab

psaab commented Aug 13, 2026

Copy link
Copy Markdown
Owner Author

Semantic merge with master edefb7570 — head is now d73835136, MERGEABLE/CLEAN

Master's #6645 REST-auth wave and this branch both changed listener identity,
credential-publish gating and management-state derivation since merge base
4960e7bee, so this was resolved from semantics rather than from which side
looked newer. One half of this PR turned out to be master's fix in a second
spelling and was removed rather than merged in.

Per-hunk resolutions

pkg/api/listener.go, the marker at serveLegLocked — master added
defer leg.drained.Store(true); the branch had nothing there. Took master's:
drained is live, pruneRetiredLocked reaps s.retiring off it.

pkg/api/listener.go, a collision git merged CLEANLY into a compile error
both sides independently added Server.HTTPSServing(), at different offsets, for
the same stated reason: Start returns a nil error when only the HTTPS bind
fails, so a caller that records a converged HTTPS fingerprint on that basis pins
itself to a listener that does not exist and never retries. The textual merge
produced two definitions of the same method. Kept one, with the branch's
strictly stronger body — httpsLeg.serving(), which adds ln != nil and
!stopping to master's != nil && !dead — and a doc comment carrying both
rounds' rationale. serving() is also what WarnStaleMgmtCertForHostName reads,
where a draining leg must not be diagnosed, so the stronger reading is the one
with a second caller depending on it.

pkg/daemon/management.go startTo/startLocked and reconcileTo's
m.srv == nil branch
— two mechanisms for one defect. #5561 round 14 (master)
and #6827 round 5 (this branch) both fixed the absorbing boot-bind failure.
Master's survives: startTo ADOPTS the api.Server whether or not the bind
succeeded, so the retry runs through reconcileTo's ordinary path. This
branch's — retain the root context, re-CONSTRUCT from the nil-srv branch — is
removed, for two reasons:

  • it would have built a fresh api.Server via api.NewServer(next), which does
    s.auth.Store(cfg.Auth) — installing the whole committed credential set on a
    freshly bound listener without ever consulting everyLiveLegNamedBy or
    publishNilDirectionLocked;
  • master's b27ab99b5 (an absent HTTP leg is not a listener at an unnamed
    address) says in its own commit message that the state it guards is reachable
    only because round 14 adopts the server. Reinstating the nil-srv early return
    would have re-stranded exactly the state that commit fixed.

m.rootCtx and startLocked are gone. The merged startTo and reconcileTo
are byte-identical to master's (verified by diffing the function bodies).

pkg/daemon/management.go add/add near publishNilDirectionLocked — pure
union; the branch's noteStaleMgmtCertHostName / deliverStaleMgmtCertDiagnosis
/ warnStaleCertForHostName and master's publishNilDirectionLocked are
independent. Both kept.

pkg/daemon/README.md — dropped the branch's "Failed BOOT start" bullet: it
documented the removed mechanism, and master's "Boot retry debt is real debt"
bullet documents the surviving one. The #6827 stale-cert-on-rename section is
untouched.

A prose claim the merge falsifiedserving()'s comment said "there is no
third defer-set flag: it would be unbindable". Master added exactly that
(drained). The comment now explains why drained is the wrong answer to
serving()'s question instead of denying it exists: it is the goroutine's LAST
act, so it reads false throughout the 5 s drain — the whole window in which the
socket is closed but the pointer is live.

Testsmanagement_recovery_6827_test.go is reduced to the two over-reach
guards that are still true and that management_bootretry_5561_test.go does not
cover (a reconcile before any start must not construct; a SUCCESSFUL boot HTTPS
bind must still record its fingerprint). The two subtests asserting m.srv
stays nil after a failed boot bind are deleted — they pinned the mechanism that
lost.

_Log.md union — structural check

Verified before this merge's own entry was prepended:

^## entries
merge base 4960e7bee 1573
branch b7272cddf 1581
master edefb7570 1587
merged 1595 = 1581 + 1587 − 1573 ✅

The header multiset also equals branch + master − base exactly (no missing
entries, no duplicates), not just the count. 1596 after this merge's own entry.

The three invariants, answered from the merged code

1. Does the credential grant still gate on where the live legs are
(5d962fcc8)?
Yes — and the premise of the question is now void, because the
listener recovery that could have rebound a leg outside the gate is gone.
reconcileTo is byte-identical to master's, so both reads of
m.cur.everyLiveLegNamedBy(next) (pre-rebind sanctioned, and the post-rebind
full-publish gate) are intact. The only remaining coupling from this branch is
HTTPSServing(), which has exactly one production call site
(management.go:242, inside startTo) where it writes m.cur.tls/httpsAddr
an input to the gate. The branch's predicate is strictly stronger
(serving()!= nil && !dead), so it can only clear the fingerprint more
often, and its two extra conjuncts are unreachable on a server api.NewServer +
Start built microseconds earlier. Master's own gate tests
(TestMgmtReconcileRevokeHonoredDespiteHTTPSBindFailure_5866,
TestMgmtNilAuthNeverDropsARetainedOffLoopbackHTTPSLeg_5561, all of
management_authsanction_5561_test.go) pass — and two of them RED under the
mutation below, so they are live guards, not vacuous.

2. Does "an absent HTTP leg is not a listener at an unnamed address"
(b27ab99b5) still hold?
Yes, and keeping master's mechanism is what makes it
hold
. everyLiveLegNamedBy's if e.addr != "" && e.addr != next.Addr guard is
unchanged. Under this branch's mechanism reconcileTo would have returned via
startLocked before ever reaching the gate, so the absent-leg state would have
been handled by a construction path with no gate at all.
TestMgmtLiveHTTPSLegIsGrantedWhenTheHTTPLegNeverBound_5561 PASS.

3. Is management state still derived from ONE committed generation
(1da594597)?
Yes. reconcile is still m.reconcileTo(m.committedDesired(cfg)).
The stale-cert fence is a diagnostic debt ledger, not a second state source:
staleCertPending/staleCertGen are read only by noteStaleMgmtCertHostName,
deliverStaleMgmtCertDiagnosis and the d.mgmt publish — never by desired,
committedDesired, m.cur or next. The identity it reports comes from
osHostname() (the kernel) read at DELIVERY, not from a config generation, so it
cannot disagree with a committed generation about the endpoint or the
credentials; its terminal action is a slog.Warn. No new lock edge into the
reconcile either: deliverStaleMgmtCertDiagnosis releases staleCertMu before
warnStaleCertForHostName takes m.mu, and reconcileWebManagement calls it
after d.mgmt.reconcile(cfg) returns. staleCertGen answers only "has another
rename landed since this delivery sampled?" — a different question from the one
committedDesired fences. TestDebtClearIsGenerationSafe_6827 PASS (both
subtests, including the negative control).

Gates

go build ./... rc 0 · go vet ./... rc 0 · go test ./pkg/api/ ./pkg/daemon/
rc 0 (TMPDIR=/dev/shm/t) · gofmt -l clean on every touched file.

Two mutations driven to RED on real assertions, build + vet clean under each,
then restored, touched and re-run GREEN:

  • widen the HTTPS-fingerprint clear to if next.TLS → reds
    TestMgmtReconcileRevokeHonoredDespiteHTTPSBindFailure_5866,
    TestMgmtNilAuthNeverDropsARetainedOffLoopbackHTTPSLeg_5561, and the retained
    over-reach guard;
  • weaken serving() back to master's != nil && !dead → reds four assertions in
    pkg/api (root_shutdown_drained_leg_is_not_diagnosed,
    reports_whether_it_reached_a_certificate,
    leg_without_a_listener_is_not_diagnosed,
    TestDrainFlagIsSetByTheRealServeGoroutine_6827) — so the stricter predicate
    is still bound after the merge rather than silently reverted.

No cluster tooling was run; nothing Rust changed.

@psaab

psaab commented Aug 13, 2026

Copy link
Copy Markdown
Owner Author

Merged with master at d73835136 — and half this PR was master's fix in a second spelling. That half is now deleted, not merged in.

I sent this to a lane as a real semantic merge rather than a _Log.md union, because master
and this branch had both changed six files since the merge base and master's changes were the
#6645 REST-auth wave — listener identity, credential publish gating, management-state
derivation. The brief said plainly that a clean textual merge would not be evidence of
correctness. It proved that concretely.

The proof, and it is the cleanest one this campaign has produced

Both sides had independently added Server.HTTPSServing(), at different file offsets,
with the same justification: Start returns nil when only the HTTPS bind fails, so a caller
recording a converged fingerprint pins to a listener that does not exist.

Different offsets means no overlapping hunk, which means git reported a clean merge — and
produced a file declaring one method twice. It did not compile. Nothing in the merge output
suggested a problem.

One survives, with the branch's strictly stronger body (httpsLeg.serving() adds ln != nil
and !stopping over master's != nil && !dead). That strength is load-bearing rather than
decorative: serving() has a second caller in WarnStaleMgmtCertForHostName, where a
draining leg must not be diagnosed.

The deletion, which is the highest-consequence call in the merge

The listener-recovery half — F2 from the round-1 hostile review — is the same defect
master's #5561 round 14 already fixed by a different mechanism. Keeping both was not an
option, and the lane kept master's:

  • The branch's mechanism re-constructed the server via api.NewServer(next), which does
    s.auth.Store(cfg.Auth)installing the whole committed credential set on a freshly
    bound listener without ever touching everyLiveLegNamedBy or publishNilDirectionLocked.

    A fail-open by construction, reached by merging two individually-correct changes.
  • Master's own b27ab99b5 commit message states that the absent-HTTP-leg state it guards is
    reachable only because round 14 adopts the server. Reinstating the branch's nil-srv
    early return would have re-stranded exactly what that commit fixed. That reasoning lives in
    the commit message and nowhere in the diff.

m.rootCtx and startLocked are gone; merged startTo and reconcileTo are byte-identical
to master's, verified by diffing function bodies rather than by eye.

The three invariants I asked to be re-checked after the combination

  • Credential grant still gates on where the live legs are. Yes — and the premise is void,
    because the recovery that could rebind a leg outside the gate is gone. HTTPSServing() has
    exactly one production call site; the stronger predicate can only clear the fingerprint more
    often. Master's gate tests pass and go red under mutation, so they are live guards.
  • "An absent HTTP leg is not a listener at an unnamed address" still holds. Yes — and
    keeping master's mechanism is what makes it hold. Under the branch's, reconcileTo would
    have returned before reaching the gate at all.
  • Management state still from one committed generation. Yes. The stale-cert fence is a
    diagnostic debt ledger, not a state source: its fields are read by three named functions and
    never by desired/committedDesired/m.cur/next, and the identity it reports comes from
    osHostname() at delivery. No new lock edge.

_Log.md union verified structurally and as a multiset — 1581 + 1587 − 1573 = 1595, zero
missing, zero duplicated. Two mutations red on real assertions with build and vet clean under
each.

What this does to the gate

The PR is materially smaller and its claims have shifted. Any reviewer verdict posted
against b7272cddf or earlier is stale on the deleted half.
A fresh hostile review is
running at d73835136, briefed to attack the deletion decision directly — if the removed
mechanism covered a case master's does not, that is a regression and I want it named rather
than assumed away.

@psaab

psaab commented Aug 13, 2026

Copy link
Copy Markdown
Owner Author

Hostile Claude at d73835136: MERGE-NEEDS-MINOR, 1 blocking — and it refutes an argument I relayed as established

First, my correction. When I posted the merge result I repeated its reasoning that the deleted
listener-recovery path "would have installed the committed credential set on a freshly bound
listener without passing the gate — a fail-open by construction." That premise does not
hold
, and the reviewer showed it by building the failure rather than reading it:

The deletion is still right, for a better reason than the one given: under master's
adopt-always startTo, the deleted arm is unreachable dead code. m.srv = srv runs
unconditionally before the error return and api.NewServer can never return nil, so after
start() has run once m.srv != nil for the reconciler's life; reconcileTo's m.srv == nil
arm is reachable only if start() never ran, and in exactly that state the branch's own guard
if m.rootCtx == nil || next.Addr == "" returns nil, because rootCtx is set only inside
startTo. Keeping it would have added a path that cannot execute.

The merge's supporting citation is accurate — b27ab99b5 does say the state it guards is
reachable only because round 14 adopts the server, so taking the branch's non-adopting
startTo would genuinely have re-stranded that fix. But the commit message should not stand
as the record of why, and I should not have relayed the fail-open half without measuring it.

B1 — BLOCKING. Both deferred-delivery call sites are unbound.

The daemon half of this PR is a debt ledger whose whole justification is that the boot rename
reaches a nil reconciler and must be retried later. Severed independently, each in its own cell:

MUT-A   delete deliverStaleMgmtCertDiagnosis() from startHTTPServer   → pkg/daemon GREEN
MUT-B   delete the delivery from reconcileWebManagement               → pkg/daemon GREEN
MUT-A+B both, leaving only the inline call in noteStaleMgmtCertHostName → pkg/daemon GREEN

With both retry points gone the mechanism collapses to "diagnose synchronously at the rename
or never"
and nothing in the suite notices.

The PR self-declares one of the two — the reconcileWebManagement site, in an explicit
"NOT YET BOUND" block — and that disclosure is honest; the reviewer confirmed it by
measurement. It does not declare the boot one, and two artefacts read as if it were
covered: a test named TestBootHostNameReachesTheDiagnostic_6827 /
boot_rename_is_diagnosed_once_mgmt_is_up, and a README sentence saying the daemon-side wiring
is "pinned by … All fail-on-revert." The subtest calls deliverStaleMgmtCertDiagnosis()
directly in its own body instead of driving startHTTPServer — so it proves the mechanism
and not the wiring.

Blocking under the rule that a test-only finding blocks when the guard is the deliverable: the
deliverable is an operator-facing diagnostic and this is its discharge path. Fix is test-only.

Everything else severed DID red — ten cells, exact assertions

serving() weakened to master's predicate reds exactly four assertions in pkg/api,
confirming the merge's count precisely. Nine more mutations red their named tests: the
d.mgmt == nil early return, the hostNameInferred gate, applyHostname's already-applied
return, the generation fence, warnCertNoSANs, the isDNSSANSafeHostname clause, the pending
flag (×3), and the noteStaleMgmtCertHostName call site. Build and vet rc=0 under every one,
so no red is a compile break.

Byte-identity verified: startTo 37/37 and reconcileTo 178/178 identical to master's, and
the whole management.go diff is two pure-addition hunks — no line inside either function is
touched. _Log.md union checked as a multiset, zero lost headers.

A second bad merge combination: searched for and NOT found, recorded so it isn't re-hunted.
Master hasn't touched three of the six "overlapping" files since the base at all; server.go's
hunks are disjoint by ~175 lines; serveLegLocked correctly carries both sides with defer
order intact; no lock-order inversion. The refactor also removed a latent panic — the old
code indexed cert.Certificate[0] with no length check.

Non-blocking

  • N3: serving()'s doc and the README both say the disable arm clears s.httpsLeg before
    retiring. ReconcileHTTPS does the reverse. The conclusion still holds — but because the
    whole switch runs under one lifeMu hold, not because of the stated ordering.
  • N4: the PR body is round-1 era. Its description of the load-path gating predicts a warning
    in exactly the case a current test asserts must be silent; it never mentions pkg/daemon,
    which is most of the production diff; and its timing evidence is 9× stale.
  • Accounting: the merge said two subtests were deleted. Three things went, and the noAddr
    half's behavioural content has no replacement — though that behaviour is master-owned, so not
    a regression here.

Fold dispatched on B1 + N3 + N4. Acceptance criteria are the reviewer's own: MUT-A and MUT-B
must go RED.

Paul Saab added 2 commits August 13, 2026 15:20
Master's #6645 REST-auth wave (edefb75) and this branch both changed
listener identity, credential-publish gating and management-state
derivation since the 4960e7b merge base, so this is resolved by
semantics rather than by taking whichever side looked newer.

DUPLICATE METHOD GIT MERGED CLEANLY

Both sides independently added Server.HTTPSServing(), at different
offsets in listener.go, with the same justification: Start returns a nil
error when only the HTTPS bind fails, so a caller that records a
converged HTTPS fingerprint on that basis pins itself to a listener that
does not exist and never retries. The textual merge produced two
definitions and a compile error. One survives, with the branch's
strictly stronger body -- listenerLeg.serving(), which adds `ln != nil`
and `!stopping` to master's `!= nil && !dead` -- and a doc comment
carrying both rounds' reasoning. Weakening it back to master's predicate
reds four assertions in pkg/api, so the stronger reading is bound rather
than merely written down.

ONE RETRY MECHANISM SURVIVES, NOT TWO

#5561 round 14 (master) and #6827 round 5 (this branch) fixed the same
defect -- a boot bind failure was absorbing, so clearing the cause never
brought management back without a daemon restart -- by different
mechanisms. Master's survives: startTo ADOPTS the api.Server whether or
not the bind succeeded, so the retry runs through reconcileTo's ordinary
path, which asks where the live legs are before it publishes a
credential grant.

The branch's mechanism -- retain the root context and re-CONSTRUCT from
reconcileTo's `m.srv == nil` branch -- is removed rather than kept
alongside it, because on top of master's startTo it is UNREACHABLE.
`m.srv = srv` runs unconditionally before startTo's error return and
api.NewServer can never return nil, so once start has run m.srv stays
non-nil for the reconciler's life. The only state that reaches the
nil-srv branch is therefore "start never ran" -- and that is exactly the
state the branch's own guard `if m.rootCtx == nil || next.Addr == "" {
return nil }` declines, rootCtx being assigned only inside startTo. An
API-disabled daemon does not reach it either: startHTTPServer runs only
under `if d.opts.APIAddr != ""`, so it leaves d.mgmt nil rather than a
nil-srv reconciler. Keeping the arm would have been dead code that reads
like a second, disagreeing retry path. And master's b27ab99 (an absent
HTTP leg is not a listener at an unnamed address) is reachable only
BECAUSE the server is adopted, so taking the branch's non-adopting
startTo instead would have re-stranded exactly the state that commit
fixed.

The removal is NOT justified by a fail-open, and that is worth stating
because the argument is tempting and wrong. Reconstructing would have
installed the committed Auth directly, seemingly bypassing
everyLiveLegNamedBy and publishNilDirectionLocked -- but master's own
recovery publishes the same full set on that path. startTo's error arm
sets `m.cur, m.curSet = mgmtEndpoint{}, false`, so on the retry
reconcile m.cur.everyLiveLegNamedBy(next) is VACUOUSLY true (e.addr ==
"" skips the first arm), the grant is sanctioned, and the pre-rebind
branch calls ReplaceAuth(next.Auth) with the whole set. The nil-Auth
case is unreachable off-loopback either way: resolveAPIBinds' #4047/#5127
clamp keys on `hasAuth := apiCfg.Auth != nil` and clamps BOTH Addr and
HTTPSAddr when it is false, so a nil-Auth construction lands on loopback
-- which is what publishNilDirectionLocked publishes there.

m.rootCtx and startLocked are gone; the merged startTo and reconcileTo
are byte-identical to master's.

A PROSE CLAIM THE MERGE FALSIFIED

serving()'s comment said "there is no third defer-set flag: it would be
unbindable". Master added exactly that -- drained, stored from a defer so
a retired leg can be reaped from s.retiring without joining it. The
comment now explains why drained is the wrong answer to serving()'s
question rather than denying it exists: it is the goroutine's LAST act,
so it reads false throughout the 5s drain, which is the whole window in
which the socket is closed but the pointer is live.

DOCS AND TESTS

pkg/daemon/README.md loses the branch's "Failed BOOT start" bullet, which
described the removed mechanism; master's "Boot retry debt is real debt"
bullet documents the surviving one. The #6827 stale-cert-on-rename
section is unchanged.

management_recovery_6827_test.go is reduced to the over-reach guards that
are still true and that management_bootretry_5561_test.go does not cover.
THREE things go, and they are not three casualties of the same kind:

  - boot_http_bind_failure_recovers_on_a_later_reconcile is DELETED. It
    asserted m.srv stays nil after a failed boot bind -- the mechanism
    that lost.
  - the noAddr half of a_disabled_api_still_does_not_construct is
    DROPPED, for the same reason: it asserted the same nil.
  - boot_https_bind_failure_retries_on_an_identical_reconcile is DELETED
    as a DUPLICATE of management_bootretry_5561_test.go, not as a
    casualty. Its own precondition asserted the OPPOSITE -- that the
    server IS adopted after a failed HTTPS bind.

The dropped noAddr half also carried behavioural content with no
replacement: after a failed boot bind, a reconcile to an EMPTY HTTP bind
must bind nothing. That still holds under master's mechanism, but only
incidentally -- startTo's error arm resets the fingerprint to
mgmtEndpoint{}, so an empty desired Addr equals m.cur.addr and the HTTP
arm sees no change. It is master-owned behaviour rather than a
regression introduced here, and it is recorded as uncovered rather than
left silently dropped.

VALIDATION

go build ./... rc 0; go vet ./pkg/api/... ./pkg/daemon/... rc 0;
go test ./pkg/api/ ./pkg/daemon/ rc 0 with TMPDIR=/dev/shm/t. Two
mutations driven to RED on real assertions, build and vet clean under
each, then restored and re-run GREEN: widening the HTTPS-fingerprint
clear to `if next.TLS` reds three tests including the retained guard, and
weakening serving() reds four assertions as above.

The _Log.md union was verified structurally BEFORE this commit's own
entry was prepended: 1595 `^## ` headers = 1581 (branch) + 1587 (master)
- 1573 (merge base), with the header multiset equal to branch + master -
base exactly.

Advances #5719.
The daemon half of #6827 is a debt ledger, and its entire justification
is that a `set system host-name` applied in the phase-4 boot config apply
reaches a nil management reconciler and must be RETRIED later. Neither
retry point was bound. Measured, each severed in its own cell: deleting
`d.deliverStaleMgmtCertDiagnosis()` from startHTTPServer left pkg/daemon
GREEN; deleting it from reconcileWebManagement left pkg/daemon GREEN;
deleting BOTH — which collapses the mechanism to "diagnose synchronously
at the rename or never" — also left it GREEN. Only one of the two was
self-declared, and two artefacts read as if the other were covered:
TestBootHostNameReachesTheDiagnostic_6827, which calls the delivery
DIRECTLY rather than driving startHTTPServer, and a pkg/api/README.md
sentence claiming "All fail-on-revert".

BOTH SITES, ONE FIXTURE, TWO OBSERVABLES

newMgmtDeliveryDaemon builds the Daemon both sites need: a REAL
configstore carrying the committed web-management stanza. That is what
made them hard to bind, in opposite directions. reconcileWebManagement
reaches its retry through committedDesired, which re-derives the WHOLE
desired state from store.ActiveConfig(); with no store the derivation
falls back to a bare Daemon's empty bind, so the reconcile DISABLES the
HTTPS leg and the delivery then correctly reports nothing served — a
green test that proves nothing. startHTTPServer reaches its delivery
through managementReconciler.start, which dereferences
d.store.ActiveConfig() unconditionally.

The two are bound on deliberately different observables:

  - the reconcile site on the DEBT FLAG. Bringing HTTPS up makes pkg/api's
    certificate LOAD path emit the same "does not cover the current
    host-name" text, so a text assertion there passes with the retry
    deleted — that is why the earlier attempt was abandoned rather than
    left in a passing-but-vacuous form. Only a delivery clears the flag.
  - the boot site on the KERNEL-NAME READ, which sits past the delivery's
    `!pending || mgmt == nil` guard and which nothing else on that path
    performs (the package's only other osHostname caller is
    applyHostname's already-applied guard). Observing it also binds the
    ORDER: an unpublished reconciler returns before the read, so moving
    the delivery above `d.mgmt = mgmt` reds the same assertion. That site
    cannot be driven end to end in-process — startHTTPServer CONSTRUCTS
    the api.Server itself and SetTLSCertDirForTest exists only after
    construction, so a serving HTTPS leg there would mean driving the
    production /etc/xpf/tls generator — and the subtest says so, with a
    negative control that the debt correctly SURVIVES a boot delivery
    that reached no certificate.

A DROPPED BEHAVIOUR RESTORED

The merge dropped the noAddr half of a_disabled_api_still_does_not_
construct along with the mechanism it was written against. Its
behavioural content had no replacement anywhere in the package: after a
FAILED boot bind, a reconcile to an EMPTY HTTP bind must bind nothing.
It is restored as a_failed_boot_then_an_empty_bind_binds_nothing, framed
as what it now is — master-owned behaviour that holds incidentally,
because startTo's error arm resets the fingerprint to mgmtEndpoint{} so
an empty desired Addr equals m.cur.addr and the HTTP arm sees no change.

PROSE CORRECTIONS

listenerLeg.serving()'s doc and pkg/api/README.md both said the disable
arm clears s.httpsLeg BEFORE retiring. ReconcileHTTPS does the reverse:
stopLegLocked first, then s.httpsLeg = nil. The conclusion survives — a
stopCh test would arm an unreachable state — but for a different reason:
the whole switch runs under ONE lifeMu hold and both serving() callers
take lifeMu, so no reader lands in between.

hostNameLikelyAccessIdentity's residual note scoped the never-diagnosed
class to boxes that drifted before this diagnostic shipped. A box RUNNING
this build reaches the same silence, because staleCertPending is
PROCESS-LOCAL: a cross-shape rename committed while nothing served a
certificate is still owed at shutdown and a restart discards it. The note
is widened in both places. Making the debt durable is declined for the
same reason as the upgrade sweep beside it — persistent state plus an
invalidation story for a name that changed again while the daemon was
down, re-firing the false positive on exactly the boxes the heuristic
cannot judge.

Daemon's stale-cert field comments are split so each field carries its
own: verified with `go doc -u`, the `Guarded by staleCertMu.` block ran
straight into the staleCertGen paragraph above a three-field group, so
godoc attached the whole thing to staleCertMu.

The merge commit message and its _Log.md entry are corrected in place.
The branch's listener-recovery arm was removed because on top of master's
adopt-always startTo it is UNREACHABLE dead code, not because
reconstructing would have failed open — master's own recovery publishes
the same full credential set on that path.

VALIDATION

go build ./... rc 0; go vet ./pkg/api/... ./pkg/daemon/... rc 0;
go test ./pkg/api/ ./pkg/daemon/ rc 0 with TMPDIR=/dev/shm/t and a
per-cell GOCACHE. Four mutations driven to RED, each in its own cell,
then restored and re-run GREEN:

  MUT-A  delete the boot delivery -> boot_start_delivers_a_debt_parked_
         before_the_reconciler_existed fails ("the boot management start
         did not attempt the parked diagnosis: the kernel name was never
         read"); the reconcile subtest stays GREEN.
  MUT-B  delete the reconcile delivery -> a_day2_reconcile_settles_a_debt_
         incurred_while_https_was_down fails ("left the host-name
         diagnosis still owed: nothing retried the delivery"); the boot
         subtest stays GREEN.
  A+B    both fail, and the rest of pkg/daemon is unaffected.
  order  move the boot delivery above the d.mgmt publish -> MUT-A's
         subtest fails.

The restored empty-bind guard is not vacuous either: making reconcileTo's
HTTP arm unconditional reds it on its no-op assertion, because api.Server
refuses to reconcile the HTTP listener to an empty bind address.

Advances #5719.
@psaab
psaab force-pushed the fix/5719-api-hardening branch from d738351 to ccc2f6b Compare August 13, 2026 22:22
@psaab

psaab commented Aug 13, 2026

Copy link
Copy Markdown
Owner Author

Round 3 at ccc2f6b09B1 closed, both sites bound independently, and the refuted argument is gone from all three places it lived

The blocker

One fixture buys both retry points, as expected — a Daemon with a real configstore carrying
the committed web-management stanza, so the reconcile is a no-op rather than a teardown. Each site
reds on its own:

MUT-A  delete deliverStaleMgmtCertDiagnosis() from startHTTPServer
       -> RED  boot_start_delivers_a_debt_parked_before_the_reconciler_existed
          "...the kernel name was never read, so deliverStaleMgmtCertDiagnosis did not run past
           its nil-reconciler guard. A host-name applied in the phase-4 boot apply is then never
           diagnosed, because the next boot's applyHostname sees the name already applied..."
       (the reconcile subtest stays GREEN in this cell)

MUT-B  delete it from reconcileWebManagement
       -> RED  a_day2_reconcile_settles_a_debt_incurred_while_https_was_down
          "...nothing retried the delivery, so the debt survives every later commit on an
           unchanged endpoint and is discharged only by another rename"
       (the boot subtest stays GREEN in this cell)

Plus an ordering cell: moving the boot delivery above the d.mgmt publish reds MUT-A's subtest,
so publish-then-deliver order is bound too.

The two observables differ deliberately, and the reasoning is the valuable part. The reconcile
site is asserted on the debt flag, because bringing HTTPS up makes the pkg/api load path emit
the same warning text — so a text assertion passes with the retry deleted. That is exactly what
killed an earlier attempt. The boot site is asserted on the kernel-name read, which sits past
the delivery's !pending || mgmt == nil guard and is performed by nothing else on that path.

An honest limit, stated in the test and both READMEs rather than left implicit: the boot site
cannot be driven end-to-end in-process, because startHTTPServer constructs the api.Server
itself and the test cert-dir hook only exists after construction — a serving HTTPS leg there means
driving the production /etc/xpf/tls generator, which an existing subtest already demonstrates
fails on permissions. So the boot subtest asserts the delivery ran with a published reconciler,
plus a negative control that the debt correctly survives a boot delivery that reached no
certificate. No "NOT YET BOUND" block remains; the disclosure is now about the observable, not
about an unbound site.

My correction landed, and was re-verified rather than relayed

The refuted fail-open paragraph is gone from the merge commit message, its _Log.md entry
(corrected in place with a dated note), and the test header. The dead-code argument replaced it —
and the round verified it firsthand rather than taking my write-up: m.srv = srv unconditional
before the error return, api.NewServer always returning a non-nil pointer, m.rootCtx assigned
only inside startTo, and startHTTPServer gated on a non-empty API address. The b27ab99b5
citation is kept, and the text now says why the fail-open reading is wrong so the next reader
does not re-derive it.

The accounting item, gone one step further than I asked

I asked for the count to be corrected. It was — and the round then checked whether the dropped
noAddr behaviour had a replacement, found none anywhere in pkg/daemon, and restored it as
a_failed_boot_then_an_empty_bind_binds_nothing, framed as master-owned and incidentally held.
Not vacuous: making reconcileTo's HTTP arm unconditional reds it.

N5 — declined with reasoning, which is the right disposition

Durable debt across a restart is not implemented, and the round says so explicitly rather than
quietly leaving the note. The reason is sound: persistent state needs an invalidation story for a
name that changed again while the daemon was down, which would re-fire the false positive on
exactly the boxes the heuristic cannot judge. The residual note now names two states instead
of one. No issue filed; I agree, and the note being accurate is what makes that acceptable.

N3 and N6 both fixed, N6 confirmed with go doc -u rather than by reading — the block had indeed
attached to the mutex rather than the three fields.

Re-gate owed at this head. Note for whoever picks it up: the head moved by force-push because the
merge commit's message was amended in place — tree byte-identical, same parents.

@psaab

psaab commented Aug 13, 2026

Copy link
Copy Markdown
Owner Author

Hostile Claude re-gate at ccc2f6b09: MERGE-CLEAN, zero blocking

And it corrected my brief on the way in, which is the right instinct: I described ccc2f6b09 as
the amended merge. It is not — it is round 3's fix commit sitting on top of it. The amended
merge is HEAD~1 = 530894153, and that is byte-identical to d73835136 (same tree
d7581e29…, same parents). So the prior review is not stale, and round 3 is exactly one commit
of delta.

The blocker, reproduced independently

Both cells red on their own subtest and nothing else in the package, each with the other
staying green:

MUT-A  delete deliverStaleMgmtCertDiagnosis() from startHTTPServer
       -> RED boot_start_delivers_a_debt_parked_before_the_reconciler_existed  (:475)
MUT-B  delete the delivery from reconcileWebManagement
       -> RED a_day2_reconcile_settles_a_debt_incurred_while_https_was_down    (:557)
MUT-A+B -> both red; neither deletion is visible to the other

Full-package runs under each confirm the new test is the unique binder for each site, so the
README's "reds its own subtest and only its own" is measured rather than asserted. The ordering
cell holds too — hoisting the boot delivery above the publish reds MUT-A at the same line.

The split-observable reasoning was tested, not accepted

Round 3 chose the debt flag over a text assertion at the reconcile site, on the grounds that
bringing HTTPS up makes the load path emit equivalent text. The reviewer verified that
empirically rather than by reading — instrumenting the day-2 subtest under MUT-B to dump the
captured warning:

msg="loaded management TLS cert does not cover the current host-name; clients verifying
     by host-name will fail — …"   PROBE_TEXT_PRESENT=true

A text assertion at that site would have passed with the retry deleted. The flag is the only
observable that distinguishes, so the choice was necessary rather than a weakening.

The boot site's kernel-name-read attribution checks out both halves: the read really is past the
!pending || mgmt == nil guard, and pkg/daemon has exactly two non-test osHostname callers,
the other being applyHostname's already-applied guard, which startHTTPServer never reaches.

Round 3 changed zero executable production code

Verified at AST level — go/parser with comments dropped, go/printer round-trip — across all
three production files. The reviewer also corrected its own first read: the raw diff looked
like a struct field reorder, but the order is unchanged; the doc comments moved between fields,
which is the attribution fix. So production behaviour is provably unchanged from the
already-reviewed d73835136, and the deliverable is entirely tests and comments.

All four links of the replacement dead-code argument verified independently, including checking
m.rootCtx against the removed code at 4120cb32c since it no longer exists in the tree. The
godoc attribution was re-checked with go doc -u rather than by reading.

N1 — non-blocking, but a real gap and worth closing

The ordering property is bound on the publish side and not the start side. Moving the
delivery to sit after the d.mgmt publish but before mgmt.start(ctx) leaves the whole
package green — and that ordering is a genuine behaviour regression: with start not yet run,
m.srv is nil, the stale-cert check returns false, and the boot retry point silently delivers
nothing at every boot.

It escapes because the subtest observes the kernel-name read, which a published-but-unstarted
reconciler still reaches. Not blocking — both sites are bound, and round 3 claims only "running
after d.mgmt is published", which is exactly what it binds. The cheap close is to have the
osHostname stub also capture whether m.srv was non-nil at read time.

N2 is a naming nit: boot_start_delivers_a_debt… reads as though the debt is discharged, while
the negative control deliberately asserts it survives.

Gate at this head: hostile Claude MERGE-CLEAN. Codex leg owed; the AGY leg is not being
counted today and I have said why elsewhere.

Paul Saab added 2 commits August 13, 2026 17:13
…peaks

Codex round 6 on the management-TLS staleness diagnostic found two runtime
defects and five assertions that could not fail on the regressions they
named.

B1, and the reason the whole debt ledger could be unpayable. An HTTPS serve
loop that terminates unexpectedly marks its leg `dead` and leaves it
INSTALLED in Server.httpsLeg — it cannot be unlinked there without taking
lifeMu, which deadlocks a shutdown racing the exit (#6401 round 3). Two
independent places then read that corpse as a converged listener. The
reconciler's leg-changed test compared only the recorded fingerprint, which
still matched the committed endpoint, so ReconcileHTTPS was never called on
any later commit; and ReconcileHTTPS's own same-address arm returned nil on
a non-nil pointer, so a call aimed directly at the configured address would
have done nothing either. HTTPS was therefore unrecoverable on an UNCHANGED
configuration for the life of the process, and every stale-cert diagnosis
went with it: the debt clears only against a served certificate, so it could
never be discharged and the restart that finally rebound HTTPS discarded it.
Fixed at both levels — the api-side no-op now tests listenerLeg.serving(),
and reconcileTo's HTTPS arm also fires on `next.TLS && !HTTPSServing()`,
which is the boot-time question of #5561 round 14 applied to the steady
state.

B2. deliverStaleMgmtCertDiagnosis emitted its warning BEFORE re-checking the
generation, so a rename landing after the kernel read logged a diagnosis
naming the host name the box had just left — the clear-side fence preserved
the newer debt but could not retract the line. The re-validation, the
certificate inspection and the clear now run under one staleCertMu hold, and
a superseded delivery abandons silently; the newer rename runs its own
delivery, so nothing is lost. Lock order is staleCertMu -> reconciler mu ->
lifeMu, and nothing under those re-enters the Daemon.

The remaining five were unbound or vacuous guards. `d.staleCertGen++` and
the unreadable-kernel-name guard were both deletable with every test green:
the race test supplied its own generation bump, and an empty name still
reaches a certificate, so the delivery reported the question answered and
cleared the debt with no identity behind it. Three assertions were written
against fixtures that make the mutation invisible — the rejected-rename cert
covered the very name a spurious note would have read, the bind-failure test
asserts a state no implementation reaches (a failed bind installs no leg
under either predicate), and the no-SAN terminality assertion used a
loopback fixture where both downstream identity checks decline anyway. The
fixtures are replaced, not the assertions, and the leg-state clauses are now
driven by the real serve goroutine rather than by flags the test sets.

Nine claim defects are corrected across the source and both READMEs:
applyHostname does not call the diagnostic synchronously; a deferred
diagnosis is not guaranteed to name the current kernel identity (Sethostname
moves the name before the generation records it, which no generation fence
can cover); only the rename entry point reads a live leg, the load path runs
before one exists; `stopping` is stored just before Shutdown, so the socket
closes an instant later and accepted requests still drain; the rename call
is the INITIAL attempt, not a retry point; the load heuristic ACCEPTS the
worked unqualified-to-unqualified shape rather than declining it; cluster
comms start after the phase-4 apply, not inside it. The restart-residual
cause list grows from two causes to seven — a dead leg, an unreadable
kernel name, an API-disabled boot, a failed HTTP start and a signal-aborted
startup all reach a restart with the debt still owed.

The boot-delivery cell now also asserts that the management server exists at
the moment of the read, which closes the hoist-above-mgmt.start escape, and
states its remaining limit in the test rather than implying more: it proves
the call happens and happens after construction, not that it reached a
certificate, warned, or discharged anything. Those three are bound at the
other retry point, where a cert-dir seam exists.

Validation: go test ./pkg/daemon ./pkg/api rc 0; go vet and gofmt clean on
every touched file; nine-cell mutation matrix in which each new assertion
has a production line whose deletion reds it and only it, including two
adjacent cells that differ (deleting the terminal `return` reds the
non-loopback no-SAN subtest and leaves the loopback one green; deleting
`d.staleCertGen++` reds only the production-increment subtest and leaves
both fence subtests green).
`go test ./...` at the round-6 head reported
pkg/refactoraudit.TestHeatmapNotStale RED: pkg/api/server.go has entered the
audit at [WATCH] (>=1500 LOC) and the committed heatmap does not list it.

The gate was already red at ccc2f6b, the head this round started from —
server.go was 1606 lines there against 1307 on master, and the heatmap is
byte-identical between that commit and this one with no pkg/api/server.go row
in either. Round 6's comment corrections added 18 lines, taking it to 1624.
Regenerating also picks up pkg/daemon/daemon_system.go at 2297 rather than
2262, drift from an earlier round of this same PR.

[WATCH] is advisory — the [REFACTOR] tier and its split obligation start
higher — so this records the tier rather than deferring a decomposition.

Validation: `bash scripts/refactoring-audit.sh` regenerated in place;
`go test ./pkg/refactoraudit/` rc 0.
@psaab

psaab commented Aug 14, 2026

Copy link
Copy Markdown
Owner Author

Round 6 at b234d0909 — all seven blockers closed, and the round corrected one of my instructions

go test ./... FULL_RC=0, whole tree.

B2 — my dichotomy was false, and the correction strengthens the finding

I told the round to "fix the ordering so the emitted name cannot be stale, or stop
claiming it cannot"
, as alternatives. The first is impossible in full: Sethostname
moves the kernel name before applyHostname records the generation, so a delivery can
legitimately read, validate and report a name the kernel has just left. No generation-keyed
fence closes that window, because the window opens before the generation exists.

So both were done: re-validation, certificate inspection and the clear now run under one
staleCertMu hold, and a superseded delivery abandons silently (round 5 warned first
and checked second) — and the "always / can never" prose is corrected in server.go,
management.go, daemon.go, both READMEs and the PR body to state the guarantee that is
actually available. a_rename_landing_mid_delivery_is_not_settled now asserts the older
delivery emits nothing.

A guarantee that cannot be made has to be written down as not made, in every place that
previously claimed it.

The mutation grid — three adjacent pairs, not one

mutation reds siblings
M4 delete d.staleCertGen++ exactly a_real_rename_mid_delivery_advances_the_generation both fence subtests green — the round-5 gap made visible
M5a drop only hostName == "" empty_name only others green
M5b drop only err != nil error_with_a_name only others green
M7 drop the no-SAN terminal return the_no_san_diagnostic_is_terminal loopback_only_identities_are_still_diagnosed green

Splitting err != nil || hostName == "" into two independently-bound halves is what turns
"the guard is covered" into "each clause is covered".

Harness honesty: the first M5 (if false {) was a build break on an unused err.
It was classified as such and not counted, then re-run in three compiling forms. A build
break scored as a red is how a matrix certifies coverage it does not have.

B3 is the round-5 gap, now visible

The previous fixture supplied the generation increment itself, inside its own osHostname
stub, so nothing could observe the production line disappear. The new cell drives the
competing rename through noteStaleMgmtCertHostName and never touches staleCertGen
building the shape where the increment is load-bearing, and M4 reds it alone.

Claim cite 6 — right finding, wrong mechanism

Codex said the load heuristic "declines precisely" the old-fwnew-fw shape. Verified
statically: it accepts it — both names are unqualified, so
strings.Contains(n,".") == qualified is false == false. The finding survives; the reason
changes entirely. The hook is needed because nothing LOADS a certificate between the boot
rename and the next restart
, and the decline applies to cross-shape renames.

A finding that is right for the wrong reason is one round from being "fixed" in a way that
does not help.

The boot-cell limitation, stated rather than papered over

Option (b) plus the cheap half of (a): the stub now captures whether d.mgmt.srv != nil at
the moment of the read, so hoisting the delivery above mgmt.start reds it. The
pre-construction seam was declined — it would put a certificate-directory field on the
production api.Config to serve one test, and the three things it buys are already bound at
the other retry point and now end-to-end by the new dead-leg test. The cell says in as many
words what it does not prove, including that a bare osHostname() substitution would
satisfy it. Same in the PR body.

Same discipline on B7: the host-name line still cannot fire on a SAN-less leaf, so the
bind-host line is the only reachable discriminator — said so in the test rather than letting
a reader infer coverage that is not there.

A red tree I handed the lane without knowing

pkg/refactoraudit.TestHeatmapNotStale was already failing at ccc2f6b09 because
pkg/api/server.go had crossed the [WATCH] threshold with no heatmap refresh. The round
measured it as pre-existing — server.go 1606 there vs 1307 on master, heatmap byte-identical
between that commit and this one, no server.go row in either — rather than assuming it
either way, then regenerated it in its own commit (b234d0909), which is the right handling
for a generated artefact.

That one is on me: I dispatched against a head without checking the suite was green there.
Noting separately that server.go is now 1624 and on a trajectory toward the 2000-line
threshold; [WATCH] is advisory and that is not this PR's problem.

Codex re-gate launched at this head.

@psaab

psaab commented Aug 14, 2026

Copy link
Copy Markdown
Owner Author

Codex at b234d0909: two MAJOR runtime problems — "I would not merge this head."

B1 — a recovered dead leg can keep serving with revoked credentials

An unexpected ServeTLS exit marks the leg dead and returns without calling
http.Server.Shutdown or Close
(listener.go:150). Serve closes the listener, but
already-accepted HTTP/1 keep-alive and HTTP/2 connections keep serving.

Recovery then installs the replacement and calls stopLegLocked(old) (listener.go:386),
pinning the old auth slot — but the old coordinator has already marked the leg drained, so
a later ReplaceAuth prunes it before tightening its credentials (server.go:855). A
surviving connection therefore continues making requests under a credential that has since
been revoked.

The recovery test cannot expose this because it accepts no connection before killing the
listener
— the same fixture shape this PR has been correcting throughout: the case is
unreachable in the fixture, not absent from production.

The unexpected-exit path must prevent further requests and keep the leg tracked until
accepted connections are gone. A bounded Shutdown alone does not force-close active
connections.

B2 — "provably current is impossible" is incorrect, and I accepted it

Round 6 argued no generation-keyed fence can make the emitted hostname provably current,
because Sethostname moves the kernel name before the generation is recorded. I accepted
that and praised it.
Codex constructed the fence:

  • acquire staleCertMu before Sethostname;
  • on success, set pending and increment the generation under that same hold;
  • unlock before persistence and delivery.

Delivery already holds that mutex across revalidation and warning, so either it warns before
the kernel changes, or the rename records first and the older delivery observes the new
generation and abandons. No lock-order inversion; a failed Sethostname unlocks without
touching the ledger.

The gap is measured: Sethostname at daemon_system.go:480, generation not recorded until
after persistence and logging at :501 / management.go:562.

This proves currency relative to every daemon-controlled rename; only an external
privileged sethostname remains unfenceable, and that residual is what should be stated
instead of a blanket impossibility.

My part: round 6's argument was that the window opens before the generation exists. The
answer is to make the generation exist before the window opens — and I did not ask whether
moving the lock acquisition earlier closed it.

B3 (minor) — two same-generation deliveries can both warn

Both sample pending=true, gen=N; the first warns and clears; the second acquires the lock,
passes the generation-only check at management.go:630, and warns again although the debt is
settled. Reachable when the boot delivery races a same-generation peer apply. Revalidation
should also require d.staleCertPending.

A round-6 adjacency claim that Codex falsified

Six of the seven mutation mappings held exactly. This one did not:

Delete leg.dead.Store(true): the unexpected-exit cell red for the asserted reason, but
both recovery cells also red
because they share that production precondition. The strict
"siblings do not red" claim is false here.

Confirmed — the listener recovery trace does work for new connections

Unexpected return closes the listener and stores dead; no automatic wake, so a later
apply/reconcile is required; the daemon condition at management.go:468 detects desired TLS
plus !HTTPSServing(); the API same-address arm at listener.go:374 also requires a serving
leg; the certificate loads, the freed address rebinds, a fresh leg installs.

So: yes on the next reconcile with unchanged configuration and without a restart; no
autonomous self-heal; and the accepted-connection lifecycle is still broken
— which is B1.

Round 7 dispatched.

Paul Saab and others added 2 commits August 13, 2026 23:05
Three findings from the round-7 review of this PR, two of them runtime.

A leg whose serve loop exits unexpectedly kept serving the connections
it had already accepted. Serve closes the LISTENER on its way out, so
nothing new arrives, but the HTTP/1 keep-alive and HTTP/2 connections
behind it stayed live under the same http.Server and the same credential
slot — while the goroutine's defer set `drained`. That flag is what
pruneRetiredLocked spends to stop tightening a retired leg's pinned
policy, on the reading that a drained leg has nobody left for a
revocation to reach. It did not. The recovery reconcile added last round
pins the dead leg's slot, the next ReplaceAuth prunes it before
intersecting it, and a credential the operator has revoked goes on being
accepted on a socket the box believes is gone.

All three exits — requested retirement, root-context shutdown, and an
unexpected serve-loop exit — now run drainLeg: a bounded Shutdown
followed by Close when the deadline expires. The force-close is not
optional. This server deliberately runs with no WriteTimeout so SSE
streams and large scrapes are not severed, which means Shutdown's
deadline has nothing behind it: an subscribed stream or a slow reader
outlives it indefinitely, and Shutdown returns ctx.Err() leaving the
connection open.

The three existing recovery cells could not observe any of this. They
drive the exit with a listener that fails from its FIRST Accept, so no
connection is ever accepted and none can survive — the case was
unreachable in the fixture, not absent from production. The new cell
binds a real loopback socket, holds one TLS connection across the kill,
the recovery and the revocation, and asserts it cannot be admitted once
the leg reports drained.

Second, the claim that the diagnosed host name cannot be proven current
was wrong, and it was mine. Rounds 5 and 6 reasoned that Sethostname
moves the kernel name before the generation is recorded, so the window
opens before any fence exists. The answer is to make the generation
exist before the window can open: renameHostNotingStaleMgmtCert holds
staleCertMu across BOTH the syscall and the bump. The two critical
sections are then ordered whichever way they race — a delivery holding
the mutex warns while the rename is still blocked, so the name it prints
is still the kernel's; a rename that gets there first has already moved
the generation, so the older delivery abandons. There is no lock-order
inversion: the fence is a leaf hold around the syscall seam, while
delivery takes the same mutex at the top of staleCertMu ->
managementReconciler.mu -> api.Server.lifeMu. What remains is a
privileged sethostname(2) from outside the daemon, which is now stated
as the residual in place of the blanket impossibility claim.

Third, the delivery's re-validation tested the generation alone. Two
deliveries for one rename — the boot delivery racing the rename's own
attempt — sample the same generation, so the first warns and clears and
the second warns again over a settled debt. It now requires
staleCertPending as well.

Validation: go test ./... -count=1 rc 0, whole tree. Mutation cells,
each run over a whole package: deleting drainLeg from the serve-exit arm
reds the new connection cell and nothing else in pkg/api; leaving the
syscall outside the fence reds the new fence cell and nothing else in
pkg/daemon; writing the ledger before the syscall reds only
failed_sethostname_is_not_diagnosed; dropping the pending re-check reds
only the new sibling-delivery subtest. One measured non-binding is
recorded rather than glossed: holding the mutex across the syscall,
releasing, and re-taking it for the bump stays green, because the gap is
a few instructions wide and the probe loses the race to the re-acquire.
That shape is still defective, so the guard against it is structural —
one deferred unlock over a body with no intermediate release — and is
documented at the function and in the test.

Also corrects a round-6 claim in _Log.md: deleting leg.dead.Store(true)
does not red one cell and leave its siblings green. Re-measured here, it
reds six across two packages, because `dead` is a production
precondition the whole dead-leg cohort shares.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The mutation that deletes the dead-leg marker reported six reds. Five of
them ran for 5.00s, 5.01s, 5.00s, 5.01s and 3.00s, and only one for
0.10s. A red sitting on a round deadline value is a poll expiring, not a
property failing, and three of these were polls on the very flag the
mutation removes.

startWithDeadHTTPSLeg polled `dead` for five seconds and then fatalled,
so the two cells that share it — one entry path between them, not two
observations — both died in the fixture. Neither ever reached
"HTTPSServing must report false" or "the reconcile left the DEAD leg
installed", which is what they exist to say. The daemon-side rebuild
cell polled HTTPSServing(), which reads the same flag, with the same
result.

The helper now joins the serve goroutine with Server.Wait instead. That
server has exactly one leg, so the join is deterministic, and it reads
nothing that is under test: when it returns, whatever the exit path
stored is settled and the caller's own assertion is what speaks. The
new held-connection cell does the same after its kill and then asserts
dead+drained explicitly, as an immediate labelled precondition rather
than a five-second wait for one.

The daemon cell cannot use Wait — its server also has a live HTTP leg,
so the join would block until shutdown — and keeps a poll, but on
HTTPSLegDrainedForTest. `drained` is stored by the goroutine's defer on
every exit path, so it reports that the exit happened without consulting
the flag under test.

Re-measured, the mutation's blast radius is unchanged, which is the
point: it is a production precondition the whole dead-leg cohort shares.
But four of the six cells now fail on their own assertions and only two
on a precondition, both immediately and both saying so.

One refinement worth recording, because the duration heuristic would
otherwise condemn a cell that is right:
TestEffectiveHTTPListenerServeExitFails reds at 3.00s and should. Its
poll IS its assertion — it waits for the listener to report Failed,
which is the property. The discriminator is whether the polled predicate
is the asserted property or a precondition for it. The duration is what
tells you to go and look.

Fixed the same fault in a cell added an hour ago: the fence test
asserted inside the syscall seam, consuming the channel value its
closing assertion then waited five seconds for, so a genuine catch
reported itself twice — once truthfully and once as "the observer never
acquired staleCertMu", which was the opposite of what had happened. It
now records in the seam and asserts after the rename returns.

go test ./... -count=1 rc 0, whole tree, 62 packages.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@psaab

psaab commented Aug 14, 2026

Copy link
Copy Markdown
Owner Author

Parent verification @ e63e7144d

CONTROL  go test -count=1 ./pkg/api/                       rc=0,  0 failures
M1       drainLeg removed from serveLegLocked's serveErr arm
                                                            rc=1,  exactly 1 failure
         --- FAIL: TestDeadLegConnectionCannotOutliveRevocation_6827

Real exit codes, not read through a pipe. Control ran in the same worktree at the same
head, so the green is witnessed.

This is a targeted mutation and it is worth saying why that matters here. One
deletion, one failure, in one package. Contrast it with the leg.dead.Store(true) cell
this PR re-measured in the same round: that one reds six cells across two packages,
because dead is a production precondition the entire dead-leg cohort is built on.
Deleting a shared precondition is closer to unplugging the machine than to probing a
wire — it proves the cohort depends on it, which nobody doubted, and says nothing about
whether any individual assertion is bound. drainLeg is the other kind: it probes one
wire, and exactly one assertion notices.

Three things in this round I want on the record

1. The vulnerability is real and the old cells structurally could not see it. A leg
whose serve loop exits keeps serving already-accepted keep-alive and HTTP/2 connections
under the same http.Server and the same credential slot, while the goroutine's defer
sets drained — and drained is exactly what pruneRetiredLocked spends to stop
tightening a retired leg's pinned policy. So a credential the operator has revoked
goes on being accepted on a socket the box believes is gone. The three existing recovery
cells drove the exit with a listener failing from its first Accept, so no connection was
ever accepted and none could survive. The case was unobservable, not merely untested.

2. The force-close is correctly identified as non-optional. This server runs with no
WriteTimeout deliberately, so SSE streams and large scrapes are not severed — which
means Shutdown's deadline has nothing behind it. A subscribed stream outlives it
indefinitely and Shutdown returns ctx.Err() with the connection still open. Bounded
Shutdown then Close is the right shape.

3. The retraction is mine and the fix is better than the framing I gave. I told this
lane in rounds 5/6 that the diagnosed host name could not be proven current, reasoning
that Sethostname moves the kernel name before the generation is recorded, so the window
opens before any fence exists. That was a false dichotomy. The answer is to make the
generation exist before the window can open: hold staleCertMu across both the
syscall and the bump. Ordered either way the race falls out correctly — a delivery holding
the mutex warns while the rename is still blocked, so the name it prints is still the
kernel's; a rename that gets there first has already moved the generation, so the older
delivery abandons. The residual is now stated honestly as a privileged sethostname(2)
from outside the daemon, in place of a blanket impossibility claim.

And the measured non-binding, recorded rather than glossed

holding the mutex across the syscall, releasing, and re-taking it for the bump stays
green, because the gap is a few instructions wide and the probe loses the race to the
re-acquire.

That shape is still defective, so the guard against it was made structural — one
deferred unlock over a body with no intermediate release — and documented at the function
and in the test. That is the correct response to a defect a racing probe cannot reliably
observe: do not pretend a test binds it, change the shape so the defect cannot be written.

Re-gate at this head.

@psaab

psaab commented Aug 14, 2026

Copy link
Copy Markdown
Owner Author

Independent hostile review @ d7157b7e1: MERGE-NEEDS-MINOR

No runtime defect — B1/B2/B3 production behaviour is correct and was proved firsthand. Two
findings, and the first one blocks on this PR.

F1 — the force-close is bound by nothing, and it is load-bearing

drainLeg's force-close (pkg/api/listener.go:154) can be deleted and
go test -count=1 ./pkg/api/ stays rc=0 over the whole package. Reverting the
retirement/root arm to the pre-round-7 bare Shutdown is also rc=0. The only thing bound
is "the serveErr arm calls drainLeg at all".

And a purpose-built probe — in-flight streaming response, listener killed under the leg —
shows the half genuinely matters: at HEAD the stream is severed (EOF); without the Close it
survives with drained=true.
That is B1 reopened through the deadline door.

This is a coverage finding, and normally coverage folds and merges. Not here: the drain
guarantee is this PR's deliverable
, and the force-close is half of it. An unbound half means
a future edit removes it and silently restores the vulnerability the PR was written to close —
with drained still reporting true, which is exactly the lie that made B1 possible in the
first place.

F2 — the justification for the force-close names the wrong mechanism

The sentence "Shutdown alone is NOT that guarantee [no connection can serve another
request]"
is false, measured rather than argued: Shutdown sets inShutdown,
doKeepAlives() goes false, and a surviving connection cannot serve a second request.

What Shutdown fails to do is terminate the in-flight response. That is the real reason
Close is needed, and it is a different claim. The conclusion is right and the reason is
wrong — which matters because the next person to read it will reason from the stated
mechanism.

The attacks that failed — the other half of the evidence

  • The split-hold "measured GREEN" claim reproduced exactly, and the reviewer notes it
    expected the opposite: sync.Mutex barging, not starvation handoff. Reporting a refuted
    expectation is worth as much as a finding.
  • drained verified stored by the defer on every path, before wg.Done — the repair from
    the earlier fixture correction rests on this and it holds.
  • No hijacker anywhere in pkg/api, so the drain has no untracked-connection hole. That
    was my first attack line and it is closed by absence, which is the right way to close it.
  • Leaf/lock-order claim verified by exhaustive grep of all three staleCertMu acquisitions.
  • B2, B3 and both new delivery wiring points each red at assertion time under mutation —
    0.00–0.03s, not deadline values. That is the three-column discipline applied without being
    asked.
  • No new HTTP/HTTPS asymmetry, no poisoned *http.Server reuse, no nil-deref on the round-6
    recovery arm.

Worktree restored clean at d7157b7e1, both probe files deleted, go vet rc=0, final control
go test -count=1 ./pkg/api/ ./pkg/daemon/ rc=0.

Round 8 dispatched: bind the force-close, correct the mechanism sentence.

Round 7 added a bounded Shutdown followed by Close on deadline, wrote
that the force-close was not optional, and bound none of it. The Close
could be deleted, and the retirement and root-context arms reverted to a
bare Shutdown, with the package still green. The only property under
test was that the serve-exit arm called drainLeg at all — so the half of
the guarantee this PR exists to provide was the half a later edit could
remove in silence, restoring the vulnerability with `drained` still
reporting true. That is the same lie that made the original defect
possible.

The new cell holds an in-flight streaming response open across each of
the three ways a leg ends and asserts it is severed. The endless handler
is load-bearing: Shutdown closes idle connections outright, so a fixture
whose handler returns leaves nothing for the force-close to do and would
pass with the Close deleted. legDrainTimeout becomes a var, unassigned in
production, so the deadline arm — which is the case under test — is
reachable without spending five seconds per subtest.

The justification was also wrong, and in a way that would have argued
for deleting the line. "Shutdown alone is not that guarantee [no
connection can serve another request]" is false: Shutdown sets
inShutdown, doKeepAlives goes false, idle connections are closed, and a
connection it is still waiting on finishes its current response and then
closes. Measured directly — a second request on a surviving keep-alive
connection fails with unexpected EOF.

What Shutdown does not do is end the response already in flight. It
waits for it, and at the deadline returns ctx.Err() leaving it open and
still streaming; with no WriteTimeout set by design, that response has no
bound of its own. Close is what ends it. Same conclusion, correct
reason, at the function, at the drained field, and in the README — the
next reader reasons from the stated mechanism, and the old one told them
this line was redundant.

drained's meaning is restated to what the drain actually provides:
nothing the leg accepted is still being served, which is both no further
request and no response in flight. Round 7 claimed only the first, which
is the half Shutdown alone already gives.

Measuring this turned up a test-design fact worth keeping in the fixture
comment: after the server severs a connection the client still returns
bytes that were already buffered — three reads and nineteen bytes here
before the error surfaced — so the assertion drains until the read
errors rather than trusting one read.

Mutations over the whole package: deleting the Close reds all three
subtests; reverting the retirement/root arm to a bare Shutdown reds those
two and leaves the serve-exit subtest green; deleting drainLeg from the
serve-exit arm reds the held-connection cell and that subtest and leaves
the other two green. go test ./... -count=1 rc 0, 62 packages.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@psaab

psaab commented Aug 14, 2026

Copy link
Copy Markdown
Owner Author

Codex @ d7157b7e1: would not merge as-is — and it corrects the hostile review

Two reviewers reached the same F1 by different routes, which raises it. But Codex also
contradicts the hostile review on a load-bearing point, and Codex is right.

The disagreement: the 5s deadline is not a wall-clock bound

The hostile review reported "5s ceiling is real … measured 5.0005s in the probe". That probe
held one connection. Codex read the paths instead:

  • the context wraps only Shutdown, and Shutdown waits for its listener group and
    synchronously closes idle connections before consulting ctx.Done();
  • after Shutdown errors, Server.Close has no context and closes connections serially;
  • for HTTPS, tls.Conn.Close may independently spend up to five seconds sending
    close_notify, per connection
    .

With an attacker-controlled connection count there is no fixed total upper bound. A
single-connection measurement cannot see a per-connection multiplier, and generalising from it
is exactly the error. The documented "bounded drain" is not bounded.

The drained invariant has an exception neither of us had

Hijacked connections. Go's Shutdown and Close both explicitly exclude them: once
hijacked, the connection leaves activeConn, Shutdown can return nil, Close is skipped —
and drained is stored while the connection remains usable.

And the sharp part: calling Close unconditionally would not fix it.

Both reviewers independently confirmed there is no Hijacker, WebSocket or custom-protocol
endpoint in pkg/api today, so this is not presently exploitable. But the hostile review
treated that absence as closure — "the drain's claim has no untracked-conn hole" — and Codex
draws the right distinction: absence makes it non-exploitable, not enforced. The
universal invariant is asserted in three places (listener.go:36, server.go:895,
README:1055) and nothing holds it. The first person to add a WebSocket inherits a false
guarantee.

Fix: either prohibit hijacking explicitly at the leg boundary, or track and close hijacked
connections there. Both are real work; narrowing the stated invariant to what is actually
enforced is the third option and needs to be a deliberate choice, not a silent one.

F1 confirmed independently, with the reason the current test misses it

requestOn drains the response body (tls_stale_cert_6827_test.go:753), leaving an idle
keep-alive that Shutdown closes without reaching its deadline. Deleting only srv.Close()
should remain green.

That is the same finding as the hostile review's F1, arrived at from the fixture rather than
from a mutation — and it explains why the suite stays green. An active SSE or a deliberately
parked handler is required.

The fence is correct; two of its claims are not

Runtime correct: production sethostname is the direct syscall seam so the hold is
leaf-like; a failed Sethostname returns before either ledger write; delivery revalidates both
staleCertPending and staleCertGen under one hold. No inversion.

But the documented three-lock nested chain at management.go:649 is inaccurate (though
conservative): the real edges are staleCertMu → managementReconciler.mu and
staleCertMu → api.Server.lifeMu, with managementReconciler.mu released before lifeMu is
acquired.

And — better than the answer this PR and I settled on — the fence test is probabilistic
despite claiming otherwise
: its observer may simply not run during the 100 ms window. Codex
proposes a synchronous Mutex.TryLock inside the syscall seam, which binds the property
deterministically. We had concluded no scheduler-based probe could land there and settled
for a structural guard plus documentation. That conclusion was right about scheduler-based
probes and wrong about the alternatives.

Two wording corrections

drained.Store(true) executes before wg.Done (defers run LIFO), so it is not the
goroutine's "last act"; and HTTPSLegDrainedForTest() can observe drained=true in the window
before Done returns, so it proves the exit body and drain completed rather than that the
goroutine has finished. Neither invalidates the tests — Server.Wait() remains the right
barrier and the accessor does not create a wrong-reason pass.

Four corrections, two of which contradict prose shipped an hour ago.
Both were verified against the go1.26.4 source before anything changed.

legDrainTimeout bounds the Shutdown, not the drain. Server.Close takes
no context and closes activeConn serially, and on an HTTPS leg each
entry is a *tls.Conn whose Close sends close_notify under its own
five-second write deadline. A peer that stalls its receive window
therefore costs up to five seconds each, one after another, so the worst
case grows with the number of such connections. "The bound this function
promises is real" was false. It now says what is bounded and what is
not, including the knock-on: Server.Wait holds lifeMu across the drain
and the stale-cert delivery waits on lifeMu under staleCertMu, so a
rename racing shutdown waits for it. Bounding the sever phase for real
needs per-connection tracking with concurrent deadlined closes, which is
a bigger change than this one; the claim is narrowed rather than the
code widened.

Hijacked connections escape the drain entirely. Go excludes them from
both calls by documented design, and a hijacked connection leaves
activeConn, so it can outlive the drain with `drained` stored and the
leg reaped. The force-close cannot help — the handle is gone. This
package has no hijacker, which makes the case unreachable rather than
handled, and the first WebSocket endpoint would inherit an invariant
that had quietly stopped holding. So the invariant is narrowed to
exclude hijacked connections, and the exclusion is now a gate: a test
walks the AST of every production file for a Hijacker assertion or a
Hijack call and fails with what drainLeg would have to grow. It reads
the AST rather than the text so that the comments about hijacking are
not false positives.

The fence cell was probabilistic while claiming it was not. Its observer
goroutine might never be scheduled inside the window it waited on, so a
pass could mean it never looked. A synchronous TryLock inside the
syscall seam answers the same question with the scheduler removed: no
goroutine, no sleep, and the answer is the state of the mutex. The cell
drops from 0.10s to 0.00s and still reds under the unfenced shape. The
split-hold non-binding now carries its mechanism rather than "a few
instructions wide": the first unlock happens in normal mode, so the
re-acquirer wins the fast-path CAS and any woken waiter re-queues, which
is why no probe of any kind lands there.

The documented lock order was a nested three-lock chain; the reconciler
releases its own mutex before calling into the server, so the real shape
is two independent edges from staleCertMu. Corrected at both sites.

Smaller ones. `drained` is not the goroutine's last act — defers run
LIFO, so it precedes wg.Done — which is corrected at the field and at
the test helper, whose answer is "the exit path and drain completed",
not "the goroutine returned". The mgmt pointer is guarded for the
stale-cert read only, and the field now says so, because the other
readers are untouched and this PR should not read as having fixed them.
drainLeg records that the arm the original defect lived in called no
Shutdown at all, so correcting why Close is needed cannot be misread as
undercutting why the drain is needed.

The reviewer asked the streaming fixture to pay the full five seconds
because the deadline is the thing under test. Split on the merits: the
serve-exit case, which is the arm the defect was in, now runs at the
production deadline so the shipped value is exercised end to end; the
other two keep the seam, since what expiry does is a property of the arm
rather than of the number; and a new test pins the shipped value so a
leaked override cannot retune production in silence.

go test ./... -count=1 rc 0, 62 packages.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@psaab

psaab commented Aug 14, 2026

Copy link
Copy Markdown
Owner Author

Round 8b @ 047c9e38d — both Codex corrections verified in stdlib source, and one of them was mine

The lane verified both corrections in the go1.26.4 source before touching anything, because
both contradicted prose it had shipped an hour earlier. Both hold.

Boundedness — the sentence was false, and so was my note

net/http/server.go:3100-3118Close takes no context and closes activeConn serially.
crypto/tls/conn.go:1471-1483closeNotify sets its own 5s write deadline. So on an
HTTPS leg a stalled peer costs up to 5s each, in series, and the worst case grows with
connection count.

My own "well inside TimeoutStopSec=20" note does not hold at scale, and the lane wrote the
accurate version rather than repeating it. Server.Wait holds lifeMu across the drain while
the stale-cert delivery waits on lifeMu under staleCertMu — that knock-on is now stated.

A real bound (per-connection tracking plus concurrent deadlined closes) is deliberately not
implemented in an eighth round; the claim is narrowed and the option written down. That is the
right disposition for work a round is not taking.

Hijack — the invariant narrowed AND the absence enforced

Shutdown "does not attempt to close nor wait for hijacked connections", Close "does not even
know about" them, and a hijacked conn leaves activeConn. So drained can be stored while such
a connection is still usable, and the force-close cannot reach it.

TestNoHijackerInThisPackage_6827 walks the AST of every production file for a Hijacker
assertion or a Hijack call — and the reason for AST is the interesting part:

a text scan would false-positive on the documentation of the rule

drainLeg's own comment and the test's own doc talk about hijacking. This is the
guard-matching lesson running inverse to every other instance on this board: nine times a
guard here has been too weak because it matched a name or a spelling; this one would be too
strong for the same reason. Match the property, never the spelling — a guard that cannot
coexist with its own documentation is not usable.

The failure message names what drainLeg would have to grow (ConnState → StateHijacked hands
you the net.Conn; track and close them there), so the warning is delivered at the point of the
change rather than in a doc nobody re-reads.

TryLock — the narrower fault was the lane's own claim

The scheduler is removed from the question: cell 0.10s → 0.00s, MUT-B2a still reds at 0.00s.
And the self-correction is sharper than the finding — the claim that failed was "the passing
direction cannot be produced by slowness"
, which it can, if the observer is never scheduled in
the window.

The shared conclusion survives exactly where it was about the split-hold shape, now
mechanistic rather than asserted: the first Unlock happens in normal mode, so the re-acquirer
wins the fast-path CAS and any woken waiter re-queues.

Also folded

Two independent edges from staleCertMu rather than a nested chain (warnStaleCertForHostName
releases m.mu before the server call). drained precedes wg.Done (LIFO), so not the "last
act". HTTPSLegDrainedForTest proves exit-path-and-drain, not goroutine-returned. d.mgmt is
guarded for the stale-cert read only, and the field says so — see #7008. And drainLeg now
records that the defect's arm called no Shutdown at all, so F2 cannot be misread as
undercutting B1.

The fixture deadline — a deliberate departure from my instruction, and it is right

I said not to shorten legDrainTimeout because "the deadline is the thing under test". The lane
split it instead: unexpected_serve_exit — the arm the defect was actually in — runs at the
production 5s; the other two keep the seam; and TestLegDrainTimeoutDefault_6827 pins the
shipped value so a leaked override cannot retune production silently.

Its reasoning, which is better than my instruction: what expiry does is a property of the arm,
not of the number
, and 15s in every future run buys the same assertion three times. That covers
what I was protecting and closes a risk I had not named. Keeping it.

Three of the corrections reported as delivered in the previous commit
were not in it. The listener.go edits went through one script that
checks every anchor and writes the file once at the end; the fourth
anchor was stale, the script raised, and none of the three earlier
edits were written. A build ran immediately afterwards and passed,
which I took as confirmation — but these are comment-only changes and
`go build` cannot see them. Checking that something succeeded is not
checking that the intended change is present.

Missing, all comment-only, all now applied and each verified by
grepping for it rather than by a build:

  - legDrainTimeout bounds the Shutdown, not the drain, with the
    mechanism (Close is serial and context-free, and each TLS
    connection's close_notify carries its own five-second write
    deadline) and the knock-on for a rename racing daemon shutdown;
  - drainLeg's hijacked-connection exclusion, which the new gate test's
    failure message points at with "see drainLeg" — so that pointer was
    dangling;
  - drainLeg's note that the arm the original defect lived in called no
    Shutdown at all, so correcting why Close is needed cannot be read as
    undercutting why the drain is needed;
  - drained is not the goroutine's last act, since defers run LIFO and
    it precedes wg.Done.

The README had all of this already, because it was edited by a
different call that succeeded. So the prose described corrections the
source did not carry, which is the divergence this PR spent a round
arguing is the dangerous one: the next reader reasons from the code.

Also adds the scope note the review asked for on the streaming cell. The
fixture holds one connection, so its bounded wait is sound there and is
not evidence of a per-leg wall-clock bound — no such bound exists. What
the cell binds is that the deadline arm severs rather than abandons,
which holds at any connection count.

gofmt and go vet clean on every touched file; go test ./... -count=1
rc 0, 62 packages.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@psaab

psaab commented Aug 14, 2026

Copy link
Copy Markdown
Owner Author

Codex @ 047c9e38d: MERGE-NEEDS-MINOR — and it refutes two things I praised

The delta from 047c9e38d to the corrected head 8b191f154 is comment-only (zero
non-comment lines in both Go files), so everything below survives that fix. Codex could not run
tests — the sandbox blocked /tmp/go-build — so these are reading findings, and they are good
ones.

1. Boundedness is still overstated, in eight more places, and even the README form is false

listener.go:131 still says legDrainTimeout bounds connection completion while
README.md:1082 says it bounds Shutdown. Even the latter is false:

Shutdown calls closeIdleConns before checking its context, and that serially closes
idle TLS connections. Each TLS close can independently spend five seconds sending
close_notify.

So the overrun is not only in Close — many stalled idle TLS peers overrun the context
inside Shutdown, and active connections then add another serial overrun inside Close. That
is sharper than anything we had.

Stale bounded-drain claims also remain at listener.go:68, :187, :270, server.go:851,
management.go:759, daemon_run_servers.go:495, README.md:1048, README.md:1243. The round
corrected the sites it knew about; this is the enumeration.

2. The hijack gate is not a semantic gate — and I called it the best decision in the round

I was right that AST beats text here, and wrong that it closes the hole. The gate detects direct
x.(pkg.Hijacker) assertions and selectors named Hijack. It misses imported handlers that
hijack internally
, with a concrete counterexample using a dependency this repo already has:

mux.Handle("GET /ws", websocket.Handler(handler))

golang.org/x/net/websocket's Server calls w.(http.Hijacker).Hijack() internally. Nothing in
the local AST matches. Reverse proxies, upgrade helpers, aliases, reflection, and connections
obtained through context escape it the same way.

The current runtime invariant still holds — ConnContext stores only peer metadata and
promhttp.HandlerFor does not take over the connection — but the claims "by gate",
"absence is enforced" and "fails if one is added" at README.md:1096 are false, and the
absolute contracts at listener.go:140, :309, server.go:895 still lack the exception.

This is the same shape the campaign keeps producing: a guard that closes the local spelling of
a property while the property is reachable from outside the scope it walks. Narrowing the claim
to what the gate actually checks — no local hijack — is honest and cheap; claiming enforcement
is not.

3. TryLock fixed the probabilistic gap and created a deterministic blind spot

It proves the mutex is held while sethostname executes. It does not prove the hold remains
uninterrupted until the generation bump — a true split hold (lock, syscall, unlock, re-lock,
bump) passes: TryLock sees the first hold and the final read sees generation 1. The test
admits this at hostname_stale_cert_6827_test.go:474, and _Log.md records B2c GREEN.

So the trade was real but not free, and it should be stated that way rather than as closure.

And Codex corrects its own earlier mechanism sentence, which I relayed approvingly: "the
re-acquirer mechanically wins and no probe of any kind can enter" is too strong — normal-mode
waiters compete, and starvation mode explicitly hands ownership to a waiter.

4. Two more overreaches, one of them in the fixture I praised

  • "Shutdown alone prevents every further request" is true for HTTP/1. HTTP/2 shutdown
    callbacks are launched asynchronously
    , so an active H2 connection can submit another stream
    before GOAWAY is processed. The final Close still protects drained; listener.go:152
    states the intermediate mechanism too absolutely.
  • assertSevered treats any read error — including its own 250 ms read timeout — as proof of
    closure.
    An open stream merely paused longer than 250 ms falsely passes. That is the
    fixture built this round to bind the force-close, and its failure mode is the one the round
    already found once: the buffered-bytes discovery fixed a single-read assertion; this is the
    same assertion trusting the wrong signal.

Confirmed

Two independent lock edges documented correctly; m.mu released before lifeMu. drained
really precedes wg.Done by LIFO — though listener.go:32 and :114 still call it the
goroutine's "LAST act". The narrowed d.mgmt comment is accurate. The historical
unexpected-exit arm truly called no Shutdown. And no test override can reach a shipped binary:
the only assignments are in _test.go with cleanup restoring them — though the default-value
test only checks the value when it runs and cannot detect an override leaked afterward into
other tests.

Three things this PR built are weaker than it claimed. All three were
checked against the source before the prose moved.

"legDrainTimeout bounds the Shutdown" was the second wrong version of
that sentence. Shutdown's loop calls closeIdleConns and only reaches its
ctx.Done select if that returns false, and closeIdleConns walks
activeConn under the server mutex closing one connection at a time. So
stalled idle TLS peers overrun the context inside Shutdown, before it is
ever consulted — the same serial-close problem the previous round found
in Close, one phase earlier. It is a poll deadline and bounds neither
phase in wall-clock terms. Swept across the eight sites the review
enumerated plus two more found by grepping the claim itself rather than
working from the list.

The hijack test is a tripwire, not a gate, and golang.org/x/net is a
direct dependency of this module whose websocket Server hijacks inside
its own handler. So a WebSocket endpoint could be added with nothing in
this package's syntax to match, and "absence is enforced" was false. The
claim is narrowed everywhere it appeared, the test grew a third check
for imports of packages known to hijack, and its own documentation now
says what still escapes: reverse proxies, upgrade helpers, aliases,
reflection.

assertSevered returned on the first error of any kind, including the
250ms read deadline it sets itself — so a stream that was merely paused,
open and free to resume, would have passed. That is the same fixture
trusting the wrong signal twice: last round it was a single read, this
round it was any error. It now treats a timeout as a reason to keep
waiting, requires a non-timeout error for closure, and names which state
it timed out in, because "still delivering" and "open but silent" are
different bugs. Re-measured under the deleted-Close mutation: still red
on all three exits, with thousands of reads, so those streams were alive
rather than quiet.

Also corrected: TryLock is a trade, not a closure — it proves the mutex
is held while the syscall runs, not that the hold is unbroken until the
bump, so the split-hold shape still passes it. And the mechanism
sentence explaining why no probe lands in that window was too strong:
normal-mode waiters compete rather than re-queue, and a mutex switches
to starvation mode after about a millisecond and hands ownership to the
waiter. The window is improbable to observe, not impossible, so that
mutant's green means "did not observe", not "cannot be observed".
Shutdown stopping further requests is qualified to HTTP/1, since h2
shutdown callbacks are asynchronous and a stream can open before GOAWAY
arrives.

go test ./... -count=1 rc 0, 62 packages.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@psaab

psaab commented Aug 14, 2026

Copy link
Copy Markdown
Owner Author

Round 9 @ 987bfe918 — all three guards were weaker than claimed, and one of the claims was mine

Codex was right on every item, each hit something this PR built, and each was verified in source
before being folded.

1. "Bounds the Shutdown" was the second wrong version of that sentence

Shutdown's loop calls closeIdleConns() and reaches its ctx.Done() select only if that
returns false
; closeIdleConns walks activeConn under s.mu, closing serially. So stalled
idle TLS peers overrun the context inside Shutdown — the same serial-close problem round
8 found in Close, one phase earlier. It is a poll deadline and bounds neither phase.

An enumeration from a reviewer is a floor, not a census — and that one is mine

I relayed Codex's eight remaining stale-claim sites as the work. The lane swept them and found
two more
(management_nilpublish_5561_test.go) by grepping the claim rather than working from
my list.

an enumeration from a reviewer is a floor, not a census

Correct, and I passed the list downstream in a form that reads as complete. A list of specific
file:line cites looks like the output of a search; it is usually the residue of a reading.
The tell: a census names its predicate; an enumeration names its members. The claim here is
a string, so the population was one git grep away. Recorded.

2. The hijack tripwire — over-read by the lane and by me

golang.org/x/net is a direct dependency (go.mod:16), so
mux.Handle("/ws", websocket.Handler(h)) is not a hypothetical — and websocket.Server hijacks
inside its own handler, defeating a local AST walk entirely.

"by gate", "absence is enforced", "fails if one is added" — false in the source, in the README,
and in my endorsement of it. Narrowed to tripwire everywhere; the test grew a third check
(imports of x/net/websocket, net/http/httputil) and its own doc now names what still escapes.

The AST-over-text decision stands and is load-bearing for a second reason. What it achieves is
"none of the three known forms is present", not proof of absence.

3. assertSevered — the same fixture, the wrong signal, twice

It returned on the first error of any kind, including its own 250 ms read deadline, so a
merely-paused stream would have passed. Round 8 fixed a single-read version of this defect;
round 9 fixed the any-error version of the same one.

It now treats a timeout as a reason to keep waiting, requires a non-timeout error for closure,
and reports which state it timed out in — STILL STREAMING versus OPEN BUT SILENT
because those are different bugs.

And it re-ran the original conclusion under the corrected instrument: deleted-Close
mutation, stricter assertion, still RED on all three exits at 3649 / 1426 / 1433 reads. The
mutant's streams were genuinely alive, so the round-8 result stands and the fixture is now honest
about why. That is the right response to discovering your instrument was wrong — re-measure the
conclusion, do not just fix the tool.

Smaller, all folded

TryLock restated as a trade: proves held-during-syscall, not unbroken-to-bump; the split-hold
shape still passes it.

The starvation-mode sentence is withdrawn — the lane shipped it on my relay and Codex was
right to retract it. Normal-mode waiters compete rather than re-queue, and sync.Mutex hands
ownership to a waiter after ~1 ms of starvation. So B2c's GREEN means "did not observe", not
"cannot be observed"
, which is a materially weaker statement than the one I passed on.

Shutdown stopping further requests qualified to HTTP/1 (h2 callbacks are async; a stream
can open before GOAWAY). Second "LAST act" at listener.go:124 corrected.

Process

Every edit this round went through a helper that writes after each replacement and reports
per-edit hits and misses, replacing the all-or-nothing script that silently dropped three edits
last round. It caught one stale anchor immediately and applied the other seven — exactly the case
that failed silently before.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant