The lifecycle layer ADR 007 assumed: one claim primitive, a timeout taxonomy, a progress heartbeat

[HCHogan]

ADR 007 built an orchestration layer on top of a lifecycle layer that does not exist. The plan engine assumes three things about the work it drives — a turn runs to completion, it is claimed exactly once, and a hang is noticed — and max implements none of them once, centrally. Each subsystem guesses again. That is why the leaks are not in the plan kernel (parser, validator, executor, reconciler are fine) but underneath it and around it.

This issue records the inventory and the prior art. It is deliberately not an ADR: the finding is that the next design decision belongs one layer below 007, and that claim should survive being wrong about the fix.

Note on method: grepping TODO|FIXME|XXX|HACK across src/ returns 2 hits, both XXX. This codebase does not mark debt; it writes debt as a well-argued decision in prose. Everything below was found by looking for structural repetition instead.

A. One primitive, hand-written seven times: claim/lease

Table	Columns	Where
episode_capture_runs	lease_owner / lease_expires_at	migrations/000_baseline.sql:726
message_deliveries	lease_owner / lease_expires_at	migrations/000_baseline.sql:1120
message_dispatches	lease_owner / lease_expires_at	migrations/000_baseline.sql:1160
maintenance_leases	whole table	migrations/000_baseline.sql:944
agent_turns	recovery_owner / recovery_claimed_at	migrations/071_durable_agent_turns.sql:21
monitors	claim_owner / claim_expires_at	migrations/073_monitors_scheduler.sql:57
plans	wake_owner / wake_claim_expires_at	migrations/077_plan_suspension.sql:35

The wait side was already factored out: Max.DB.Notify has a WorkChannel enum and five channels (max_dispatch_work, max_delivery_work, max_monitor_work, max_plan_work, max_timeline_work) sharing one claimOrWait. The claim side never was. The abstraction is half-built, and the half that was left out is the half with the invariants.

The cost is not aesthetic. ADR 007 shipped the seventh copy and got it wrong the same way: a two-column desired/running diff re-dispatches finished children forever, which is what reconciled_revision in migrations/078_plan_concurrency.sql exists to fix. Six prior implementations had no shape in which that lesson could be carried.

B. Timeouts exist at exactly one layer, and it is the innermost one

Layer	Ceiling
A single LLM call	yes — 120s/300s, per profile
The agent loop	rounds only: maxTurns = 200 (src/Max/Effects/Agent.hs:173)
A front-model turn	none — src/Max/Handler.hs:2148 calls agentTurn bare
A fork child	yes — race … (threadDelay …) (src/Max/Handler.hs:1997)
A tool call	each tool decides for itself
Acquiring a DB connection	none — Data.Pool.withResource has no acquire timeout

src/Max/Effects/Agent.hs:161 states the split outright: "timeouts are configured at the 'LLM' layer; these are loop-level." Nothing owns wall-clock for the turn. Worst case is 200 rounds × 300s.

The pool is the only genuinely unbounded wait in the process, and it is process-wide rather than per-conversation: src/Max/DB/Connection.hs:26 builds one 24-resource, single-stripe pool, four of which are permanently held by LISTEN waiters.

C. absorb is a substitute for a queue that was never built

One turn at a time per conversation is right. Merging the second message into the first is the part to revisit. The predicate at src/Max/Handler.hs:1873 is listTasks … (Just gm.groupId) — conversation only, not author, not topic. Production shows it firing across authors:

01:10:02 dispatch/llm absorbed into a running turn  aimed=false group_id=650536599 message_id=98421 task=t2

aimed=false, and a different sender than the one whose turn was running. The code is honest about it (NoteAmbient, "nobody claimed this message is about the work") but the consequence is that an unrelated question becomes ambient noise in someone else's context and never gets its own answer.

Combined with B this is the real outage shape: a stuck turn silently swallows every subsequent message in the conversation, each acknowledged with the 托腮 reaction whose comment promises "the message will be read" — a promise that is false exactly when it matters.

D. Debts owed by a dead turn are not durable

completeDispatch … DispatchCompleted runs immediately after the async fork (src/Max/Handler.hs:523), before the turn has computed anything. Absorbed messages live only in the in-memory inFlightTriggers. Kill a stuck turn and the questions it swallowed are not re-dispatched — they are answered only if a later message happens to start a turn that reads them out of history.

E. The orchestration layer leaks at its own exit

• A fork child's tool ceiling is the plannable catalog — web_search, get_message_by_id, context_search, memory_list (src/Max/Plan/Catalog.hs) — because childGrants intersects against planCatalog (src/Max/Handler.hs:1258). ADR 007 §"The front model keeps the fast tools" (line 553) says the opposite: "the split is by latency, not by capability: Browser, Sandbox, Video … belong in children." The catalog's admission rule is "someone hand-wrote ptResult", which the plan expression language needs and a child does not — a child's typed boundary is subgoal_return, whose schema is the goal's expected type. Conflating the two makes fork buy nothing but parallel web_search.
• WatchEach / JoinAll are parsed and never consumed: no reference outside Max.Plan.Types and Max.Plan.Parse.
• Nothing but the front model can open a plan. No command, no monitor, no operator entry point.

F. Fifteen workers, no shared failure policy

Max.Worker is the whole supervisor: a required worker returning normally throws, and every worker exception is linked to the parent (src/Max/Worker.hs:59). No restart, no isolation. So "what counts as fatal" is decided independently inside each of the fifteen workers — matrix swallows its 504s, imessage swallows its timeouts, each by hand.

Prior art worth borrowing

Temporal's timeout taxonomy (activity execution, detecting activity failures) is the most directly transplantable piece. Four timeouts, each catching a different failure: schedule-to-start (worker dead or fleet behind — Temporal defaults it to ∞ and recommends a metric instead), start-to-close (strongly recommended; the server cannot detect a silently wedged worker, so it depends on this to force a retry), schedule-to-close (total across retries), and heartbeat (for long-running work: a periodic ping meaning "still making progress"; no ping inside the window means the worker is presumed dead).

max is one field away from the heartbeat: setTurnPhase already fires at both round boundaries (src/Max/Effects/Agent.hs:348 for "llm", :430 for "tools") but writes only the phase text (src/Max/Tasks.hs:337). Stamping a time there yields a real progress clock that the absorb predicate, !ps, and a watchdog can all read; TaskInfo currently exposes only tiStartedAt (src/Max/Tasks.hs:198), which is start age, not progress.

Orleans/Dapr virtual actors (actor features, reentrancy) are what C should become. Turn-based concurrency takes a per-actor lock and queues the rest; max takes the same lock and merges the rest. Worth noting that ADR 007 already invented the other half independently — "a child is not decided while a plan it opened is suspended" is reentrancy. The queue is the missing piece, not the lock.

FOR UPDATE SKIP LOCKED (pattern, caveats) is the shape the seven copies in A should collapse into: one claim function, a (status, ordering) composite index per queue, claim committed with the transaction.

Cognition, Don't Build Multi-Agents is the honest counter-reading to keep on file. Its argument — isolated parallel subagents make implicit decisions under diverging assumptions and produce work the coordinator cannot reconcile — does not refute ADR 007, whose justification for Fork is latency, not intelligence. It lands squarely on the JoinAll gap: a fork whose combining step cannot be written in advance is a net loss, which ADR 007's own Consequences section already concedes. Read as evidence for splitting by latency and against splitting by task.

Work

• One claim primitive. SKIP LOCKED + owner + expiry, beside the existing Max.DB.Notify wait side, with the seven call sites in A migrated onto it. Landing this first is what makes the rest cheap to state.
• A timeout taxonomy. start-to-close and heartbeat, declared separately at the turn, child and tool layers. The front-model turn is the missing one; note it cannot copy the child's race, because a front turn has already streamed part of a reply and must run an epilogue (hold what was sent, swap the reaction, requeue the inbox) rather than being dropped.
• A progress heartbeat. Timestamp setTurnPhase; expose it on TaskInfo; read it from the absorb predicate, !ps, and a watchdog.
• Bound the pool acquire. The one unbounded wait in the process. Failing to get a connection should be an error, not a permanent stall.
• Queue instead of absorb. A second message in a live conversation queues behind the running turn; merging becomes the explicit reentrant case (same author, or an explicit reply into the running turn), not the default.
• Make an owed answer durable. completeDispatch should not mark a row done before the turn it spawned has produced anything, so a killed turn returns its debts to the queue.
• Widen the fork child's ceiling. Translate tdEffects → PlanEffect and intersect against the goal budget, instead of intersecting against planCatalog. This is what makes Fork mean what ADR 007 §553 says it means. Blocked on nothing above, but pointless until B and C stop a slow child from wedging its parent conversation.

Sequencing: claim primitive → timeout taxonomy + heartbeat (the heartbeat is what the timeouts are measured against) → queue-instead-of-absorb and durable-owed-answer (both read the lifecycle state the first two establish) → child ceiling. The pool acquire bound is independent and can land at any time.

[HCHogan]

A (claim primitive) — first half landed in 6e7c35a.

The primitive turned out to belong in SQL rather than in Haskell. The seven copies differ in their payload joins and their eligibility predicates — both irreducibly domain-specific — but they were also each writing out the lease test by hand, and that is where they had drifted:

axis	spellings found
what makes a row free	null owner (monitors, plans) vs null expiry (dispatch, delivery)
boundary	< (dispatch, delivery) vs <= (everything else)
whose clock	server now() vs a timestamp the caller computed

The third is a live defect, not untidiness: monitor_fires and plans compared the stored expiry against the caller's clock, so a worker running fast reclaimed a lease that had not lapsed and both owners drove the same plan.

max_lease_free(owner, expires_at) and max_lease_until(seconds) state it once (migration 081). Migrated: message_dispatches, message_deliveries, monitor_fires, plans.

Deliberately not migrated, each for its own reason:

• agent_turns stores recovery_claimed_at, not an expiry — needs a schema change first.
• maintenance_leases is already server-clocked and is the only one of the seven with a fencing token; plans gained one separately via wake_epoch. The other five have none, so an expired owner can still complete its work. That is its own finding and probably its own item.
• episode_capture_runs folds the expiry sweep into the candidate predicate rather than running it separately.

Still open in A: the setter side (writers still compute the expiry client-side; max_lease_until exists for them), and the expired-row sweep, which differs by target state across tables and did not unify cleanly.

Verified by mutation — replacing the predicate body with SELECT true on a fresh database fails two lease tests — and rehearsed against a schema-only copy of production, which sits at 080.

[HCHogan]

B (timeout taxonomy) and the heartbeat — a08eb3c + 46682cc.

The heartbeat first, since the ceiling is measured against it. setTurnPhase already fired at every round boundary and threw the timing away; every writer of teKind now goes through one writePhase so phase and heartbeat cannot disagree about when a turn was last seen. Exposed as tiProgressAt, read by !ps (as idle=, only past 30s) and the admin task JSON. Granularity is the round, not the tool call — a round carrying three slow tools stamps once, at its start.

Then the ceiling, and it is not start-to-close. Age is what a legitimately long research turn also reports, so a total budget would have to be set high enough to be useless against the case it exists for. Silence is only ever reported by a turn that has stopped. awaitTurnSilence restarts its wait from the heartbeat TVar itself (orElse against registerDelay, no polling), so honest multi-round work pushes its own deadline out and a wedged round is noticed once, exactly at the deadline.

Raced rather than checked at a round boundary — the failure this exists for is reached from inside a round, where a boundary check never runs. The price is the loop's bookkeeping, so the timeout branch settles the turn the way the existing no-reply branch does (swap the processing reaction for the failure face, drain nothing) rather than inventing an AgentResult. Text that already streamed is already in the group; the face is what says it was cut off rather than finished.

Default turn_silence_seconds = 600, derived rather than picked: the longest legitimate single phase is a tool round, where sandbox_exec alone accepts up to 300s and a slow LLM profile another 300. That is the honest floor at round granularity — tightening it means stamping per tool call, which is its own change and worth doing before this number gets treated as tight.

Still open in B: nothing bounds an individual tool call (only web_search and sandbox_exec carry their own), and the DB pool acquire is still unbounded — the one wait in the process that no ceiling covers, and process-wide rather than per-conversation when it bites.

Not done, deliberately: the absorb predicate still does not read tiProgressAt. It belongs with C (queue instead of absorb) rather than bolted onto the ceiling, since both want the same threshold decision made once.

[HCHogan]

C (queue instead of absorb) and D (durable owed answer) — 5d12f25. They had to land together: deferring a message is pointless if the row saying it is owed has already been marked answered.

C. Two things say a message belongs to work already in flight, and only they: it replies into that turn (or into something the turn already swallowed), or its author is the person whose question the turn is answering. That is the Orleans/Dapr reentrancy rule — the per-conversation lock stays, the merge goes. Anything else is deferred. Non-direct origins (proactive, poke) keep the old behaviour outright: no durable row exists to come back to, so folding in is the only place they can go.

The queue already existed — this table, with its lease, its next_attempt_at and its NOTIFY trigger. What was missing was a way to say "not now" that an operator reading the failed rows does not have to sort out. 'deferred' is deliberately absent from max_notify_dispatch_work: a deferred row is scheduled, so waking a worker that will find next_attempt_at in the future is a wasted round trip. The precise wakeup is releaseDeferredDispatches, which writes the row back to pending when the turn ahead ends and thereby fires the existing trigger. The 30s retry is the bound for a row that deferred itself in the window between that release and the turn leaving the registry — a residual race, named rather than papered over.

D. runDispatchClaim marked the row completed the instant the dispatch async was forked, before a token had been spent. The turn now owns its row and settles it on every exit through the finally; the claim loop settles only what it finished itself. A process that dies mid-turn leaves the row claimed, which the existing sweep quarantines to outcome_unknown — worse than an answer, better than a lie.

Ordering is a fact, not a hope: the deferral and the epilogue's unconditional completion are written by the same thread in that order, and completeDispatch only matches a row still claimed by this worker, so the first write decides. There is a test that asserts exactly that.

Rehearsed against a schema-only copy of production (99k dispatch rows, 11 MB — the new partial index is milliseconds) and every existing status is inside the widened constraint.

Not done, and worth stating. The absorb path still does not consult tiProgressAt: a turn that has gone silent can still absorb an aimed reply for up to turn_silence_seconds. Guarding it interacts with the one-turn-per-conversation invariant — refusing to absorb means starting a second concurrent turn in that conversation — so it wants its own decision rather than being bolted on here.