Skip to content

feat(automation): once-per-tick-window dispatch claims for scheduled flows - #16585

Merged
os-trump merged 12 commits into
mainfrom
claude/issue-14501-schedule-dispatch-claim-ledger
Sep 7, 2026
Merged

feat(automation): once-per-tick-window dispatch claims for scheduled flows#16585
os-trump merged 12 commits into
mainfrom
claude/issue-14501-schedule-dispatch-claim-ledger

Conversation

@os-trump

@os-trump os-trump commented Sep 7, 2026

Copy link
Copy Markdown
Collaborator

Fixes #14501

Behaviour half of the maintainer's A + a2 ruling (decision batch #13, issuecomment-5518058223, 「同意」). The contract half landed in packages/spec via #14766 / PR #15377; this PR only consumes it, and mints no new error code.

🔴 Clause-② is yes. An earlier revision of this body derived no from zero packages/spec paths in the diff — a path test standing in for a surface test. Clause-② asks whether the change widens the accept set or the public surface, and this one does: new exported symbols on published barrels, plus two new published columns on sys_flow_dispatch and a deliberately retired row-immutability invariant. Corrected here for the audit record. The isolated contract review at CONTRACT_REVIEW_TIER has run twice; this revision is its second, text-only patch round.

What the ruling asked for, and where each piece landed

A schedule (cron) flow now takes a persisted (flow, tick-window) claim in the same sys_flow_dispatch ledger a time_relative flow claims per (flow, window, record) (#10220), and settles that claim with what the run turned into.

ruled behaviour site
(flow, tick-window) claim before launch, no-op on a hit packages/triggers/trigger-schedule/src/schedule-trigger.ts
the claim records its outcome service-automation's FlowDispatchStore + sys_flow_dispatch
replay() refuses a delivered window, force is the door packages/services/service-job/src/db-job-adapter.ts
the ticker still survives a throw unchanged swallow, now settling the claim failed

The specification read for this was IJobService.replay's TSDoc decision table on origin/main, not the issue thread.

The measurement the dispatch asked for: claim() returns a boolean, and that was not enough

FlowDispatchStore.claim(key) answers boolean, and both the store's doc-block and sys_flow_dispatch's own header state the rows are immutable claims — the engine slice deliberately declared only find and insert, so that a double could not be looser than the writer. The ruling's "the claim records its outcome" cannot be expressed in that shape: a row's existence says only that someone took the key, never whether the dispatch landed.

So this is a real, declared schema and interface change, stated rather than smuggled:

  • sys_flow_dispatch gains two optional columns, outcome (succeeded / failed) and settled_at. The row is now two-phase: claim() still writes it before the launch and still wins the race on the primary key, and settle() is the only writer that ever updates one. The object's header records that this is the one place the immutability stopped, and states the write rule exactly — succeeded is absorbing:

    from to
    null succeeded / failed an ordinary run settling its own claim
    failed succeeded required — a replay repaired the window, so the next unforced replay must be refused
    succeeded failed refused

    The refusal is not tidiness about monotonicity. The ruling guarantees that a window whose claim succeeded is refused, and "latest attempt wins" erodes exactly that: a forced replay that throws would rewrite a delivered window to failed and silently reopen the unforced re-delivery door — the duplicate-delivery harm this card exists to close. An operator whose forced replay failed forces again, which is louder and safer than a door that reopens itself. The predicate is isSettleAllowed, and refusing is a no-op rather than a throw: it is the invariant working, not an error.

    ⚠️ Against the persisted store the rule is enforced read-then-write, so it is not atomic, and the settle() docblock now says so. A succeeded and a failed for one window finishing at the same instant — a replay and a tick, or two replicas — can interleave and leave the row failed. The reach is narrow and the engine's by-id update shape cannot express a conditional write, so the window is named rather than closed; the in-memory store, whose check and write share a turn, does not have it. Closing it needs a conditional update the engine does not have yet.

    ⚠️ An earlier revision of this body, and of both headers, said settle() writes "only from null to a terminal value". That was false about the very transition this PR adds — the trigger settles after every run, including a replay pass, so failed to succeeded was always reachable. Both headers and this body now state the rule as implemented, and the transition is pinned in both directions.

  • FlowDispatchStore gains optional settle() and read(). A host supplying its own store keeps working: dedup is unchanged, only the replay refusal is lost, and the engine says so once.

  • FlowDispatchStoreEngine now requires update — which is why every double in flow-dispatch.test.ts routes through assertEngineUpdateDispatch.

⚠️ That gate caught a real defect before it shipped. The first implementation wrote update(object, id, data, options). No engine here dispatches on that four-argument form; the canonical shape is update(object, { id, ...fields }, options). check:engine-double-contract refused the loose double, and the producer was wrong, not the gate.

A row left at outcome: null — a process that died mid-launch, a row written before this PR, or a time-relative: claim, which is never settled at all — reads as not delivered, which is the replay() contract's "failed" row: it re-runs rather than being refused on a claim nobody settled.

The window key

There is exactly one notion of "window", derived from the schedule descriptor itself, so the tick that takes a claim and the replay that reads it back cannot disagree.

  • cron — the previous occurrence of the very same expression in the very same timezone, computed with croner, the library the job adapter already schedules with. A DST or leap boundary therefore moves the fire and the window key together by construction; there is no second calendar to disagree with, which is what the dispatch asked to be checked. croner resolves to whole seconds and answers strictly-before, so the reference is advanced to the start of the next second to make that at-or-before. ⚠️ That property rests on an assumption now named in the TSDoc: an absent schedule.timezone falls back to 'UTC' here, the same default CronJobAdapter applies to the fire. A host that constructed its adapter with a different default timezone would fire on its calendar while this keys on UTC.
  • interval — the epoch-anchored bucket floor(now / intervalMs). Anchored to the epoch, not to registration time: a restart re-registers the timer at a new offset, and a window that moved with it would forget every claim across exactly the restart this ledger exists to survive.
  • once — the single instant the job is due, which is one window for the job's whole life. The visible consequence is on replay: an operator who replays a one-shot job before its due instant claims that single window, so the real fire then finds the claim and does nothing. Previously both ran. ⛔ Whether a once job re-fires at all after a restart is not measured here and is not asserted.

⚠️ The dispatch's suggested route was measured false and not taken. There is no "window identity the cron schedule already computes for its fire": CronJobAdapter fires runScheduled(name) with the job name only, and JobHandler's context is { jobId, data } — no deadline reaches the handler. The window has to be computed from the descriptor, which is why croner moves from a devDependency to a dependency of @objectstack/trigger-schedule (it is already a runtime dependency of @objectstack/service-job, same version range).

Verification

Every pin was written with its prediction recorded first, then proven able to fail by ablation — mutate, prove the mutation reached disk, run, restore, prove the restore by blob hash and a whole-tree git status --porcelain.

ablation predicted observed
drop the claim-miss return in the trigger 3 red 3 red, the named three
drop the refusal in DbJobAdapter.replay 4 red 4 red, the named four
drop the failed-outcome settle 1 red 1 red, the named one
rethrow from the error-isolation catch 2 red 4 red — see below
make isSettleAllowed always true (own suite) 3 red 3 red, the named three
the same, reached through service-automation's built dist 1 red 1 red, the named one

The rethrow prediction was written against an earlier harness. After the pins were split by subject (below), the throwing-run tests no longer route through an adapter, so a rethrow propagates in four places rather than two. Direction as predicted, count higher; recorded here rather than rewritten.

⚠️ One ablation leg was thrown away rather than reported. The first run of the absorbing-rule dist leg answered exit 1 with No test files found — the runner had used one package for both the rebuild and the suite, so the trigger's suite was looked for inside service-automation. That is NOT MEASURED, not a red: it was discarded, the runner was fixed to take the test package separately, and the leg was re-run against the right suite with the marker verified live in dist first (ablation-dist-preflight.mjs, marker present in 2 built files; absent again after restore).

⚠️ A regression this PR caused and fixed on the merits. The new "no replay guard could be installed" warning broke kernel-rebuild-rebind.test.ts's expect(logger.warns).toHaveLength(0) — a pin whose whole point is that a bind failure must not degrade back to a WARN. The pin was not weakened. The warning is now emitted only when a ledger is actually attached, because with no ledger nothing is ever recorded as delivered, so there is no refusal to lose and the line was a false alarm.

Public surface, narrowed to what has consumers. computeTickWindow, scheduleDispatchKey and TickWindow are not on @objectstack/trigger-schedule's barrel: measured 0 consumers outside the package, they appear in no exported signature, and the in-package users import them from ./schedule-trigger.js. They stay exported from that module. Symmetrically, ReplayGuard and ReplayGuardDecision are now exported from @objectstack/service-job's barrel — DbJobAdapter.setReplayGuard(name, guard) is a public method on a barrel-exported class, so leaving its parameter type unnameable from that package is the asymmetry the repo's own nameability precedent forbids.

Pins split by subject, at a gate's own instruction. Reaching DbJobAdapter from trigger-schedule's tests needed a new entry in scripts/check-test-source-alias.mjs's registry, which is shrink-only, and the paths route is foreclosed by that package's rootDir of ./src. check:type-source-resolution names exactly that case and says to reach the subject through in-package source instead. So the adapter's refusal, its force door and its guard protocol are pinned in packages/services/service-job/src/db-job-adapter.replay-guard.test.ts, and the trigger's side pins the guard it actually registers.

Gate family, derived mechanically on the final head 8e08371e2 with node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack: 103 derived, 103 run, 0 NOT-MEASURED, 0 UNRUN, 0 red, reconciled with --ran. Each exit code was captured immediately after its own single redirection, never through a pipe. The family grew 67 → 103 once the census regeneration put content/docs in the change set; the docs families were run too. Every gate that answered exit 3 PREREQUISITE NOT MET, and the spec docs gates that needed generated artefacts, was re-run for real after a full workspace build (72/72 tasks, exit 0) rather than being read as a pass. ⚠️ Three gates that read built output were also once measured while a workspace build was still in flight; those results were discarded rather than reported, and the whole family was then re-run against the finished tree, which is the sweep quoted here.

Suites: service-automation 1476 ✓ · service-job 117 ✓ · trigger-schedule 82 ✓; typecheck green for all three.

验收备注

Out of scope, noted and not filed:

  • IntervalJobAdapter.executeJob swallows a handler rejection too and records it as a failed execution. So a scheduled flow's throw passes through two swallows, not the one the card names — the trigger's, and the adapter's. It changes nothing for this ruling (the claim now records the failure either way) but it is why the ticker pin is asserted on the handler the trigger registers rather than on a fire routed through the adapter: through the adapter, the pin would be green against a trigger that rethrows.
  • An abandoned replay pass is cleared only by this flow's next guard call or by stop() — a later fire does not clear it, because the handler deletes the entry only when the pass matches the window it just computed. It stays inert regardless: a pass naming a window that has passed can never match again. An earlier revision of that comment said a later fire clears it, which was simply wrong.
  • content/docs/permissions/tenant-audit-census.mdx and its counts file move 222 → 223 write call sites, with the hand-written prose figures updated alongside. That is the ledger's settle() write being counted, regenerated with scripts/tenant-audit-census.mjs --write.

Generated by Claude Code

os-trump and others added 7 commits September 7, 2026 10:06
A `schedule` (cron) flow now takes a persisted `(flow, tick-window)` claim in
the same `sys_flow_dispatch` ledger a `time_relative` flow claims per
`(flow, window, record)`, and settles that claim with the run's outcome. That
closes the two live duplicate-delivery doors: a restart inside a tick window,
and an operator `IJobService.replay()` of a window that was already delivered.

`DbJobAdapter.replay` refuses a delivered window with the ADR-0112
`RESOURCE_CONFLICT` / 409 envelope the contract declares, and `{ force: true }`
is the door past it. The error-isolation catch in the schedule trigger keeps
protecting the ticker exactly as before; it stopped being silent, recording the
throw on the claim as a failed outcome instead of leaving the run
indistinguishable from a delivered one.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012zTkyNHJ7TkuN2oXtP5x37
…it the pins by subject

The ledger's settle() wrote update(object, id, data, options); no engine
dispatches on that four-argument form. It is now the by-id payload shape
(update(object, { id, ...fields }, options)) the ObjectQL engine takes, and the
test doubles route through assertEngineUpdateDispatch so a fake can never again
be looser than the engine it stands in for.

The DbJobAdapter.replay pins move to service-job's own suite: reaching them from
trigger-schedule needed an entry in the shrink-only test-source-alias registry,
and that package's rootDir forecloses the paths route.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012zTkyNHJ7TkuN2oXtP5x37
A pass a job service asked for and then abandoned no longer outlives its
window: at most one outstanding pass per bound flow, dropped with the binding.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012zTkyNHJ7TkuN2oXtP5x37
…ngine predicate

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012zTkyNHJ7TkuN2oXtP5x37
@github-actions

github-actions Bot commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

This PR changes 3 package(s): @objectstack/service-automation, @objectstack/service-job, @objectstack/trigger-schedule, touching 40 documentable anchor(s). ⚠️ 4 changed file(s) yielded no anchor (packages/services/service-automation/src/index.ts, packages/services/service-job/src/index.ts, packages/triggers/trigger-schedule/package.json, …), so the pages documenting them are NOT COVERED by this run — this is not a clean bill of health for those files.

6 hand-written doc(s) NAME something this change touched and may need an implementation-accuracy re-verification:

  • content/docs/api/error-catalog.mdx (via RESOURCE_CONFLICT (literal, a string literal in replayConflict))
  • content/docs/api/error-handling-server.mdx (via RESOURCE_CONFLICT (literal, a string literal in replayConflict))
  • content/docs/data-modeling/objects.mdx (via highlightFields (symbol, a field of const object SysFlowDispatch))
  • content/docs/data-modeling/schema-design.mdx (via highlightFields (symbol, a field of const object SysFlowDispatch))
  • content/docs/deployment/cli.mdx (via highlightFields (symbol, a field of const object SysFlowDispatch))
  • content/docs/ui/forms.mdx (via highlightFields (symbol, a field of const object SysFlowDispatch))

4 release-owned page(s) also name something this change touched. These are read-only:

  • content/docs/releases/v12.mdx (via highlightFields (symbol, a field of const object SysFlowDispatch))
  • content/docs/releases/v15.mdx (via highlightFields (symbol, a field of const object SysFlowDispatch))
  • content/docs/releases/v16.mdx (via AutomationEngine (symbol, a top-level class))
  • content/docs/releases/v17.mdx (via AutomationEngine (symbol, a top-level class))

content/docs/releases/ is RELEASE-OWNED (AGENTS.md "Documentation Guardrails"): release
notes are written centrally at release time, and a code PR that edits them is the exact PR
that guardrail exists to stop. They are still audited — read-only. If one of them is actually
wrong, file an issue or open a dedicated docs-only PR; do not edit it here.

What this run could not see
  • 4 changed file(s) yielded no anchor (packages/services/service-automation/src/index.ts, packages/services/service-job/src/index.ts, packages/triggers/trigger-schedule/package.json, …) — pages documenting those are invisible to this run
  • 16 name(s) were too generic to anchor anything (single lowercase words)
  • the SDK route bridge reached 61 of 219 client-bound route-ledger rows — the other 158 have no registrar path: tail to select them, so pages documenting THEIR client methods cannot appear above, on this or any run. Of those 158: 0 are remediable by widening that discovery convention (an in-repo file declares the path; the convention did not scan it); 56 are structural — on a ledger where NOT ONE row is declared in-repo, so no discovery change reaches them at any price; 102 are undecided (no in-repo declaration, on a ledger that has other in-repo registrars — absence and an unreadable spelling are not distinguishable here). The rows themselves: node scripts/docs-audit/affected-docs.mjs --bridge-coverage
  • a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.

Coarse fallback — 6 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json a7c14cdf371bdb0f22be28e47a40fab9f7be88d0packageMentionDocs.

Which tree this was computed on

This run read content/docs from 8c84f4f808743f4a0d6fe0e899d69d378f6bfaba — the merge of head 8e08371e2509d340075a9b9d3a706ad7356d313d into base a7c14cdf371bdb0f22be28e47a40fab9f7be88d0, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 8c84f4f808743f4a0d6fe0e899d69d378f6bfaba && git checkout 8c84f4f808743f4a0d6fe0e899d69d378f6bfaba
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin a7c14cdf371bdb0f22be28e47a40fab9f7be88d0 8e08371e2509d340075a9b9d3a706ad7356d313d && git checkout -B drift-repro a7c14cdf371bdb0f22be28e47a40fab9f7be88d0 && git merge --no-ff 8e08371e2509d340075a9b9d3a706ad7356d313d

node scripts/docs-audit/affected-docs.mjs --json a7c14cdf371bdb0f22be28e47a40fab9f7be88d0

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

Advisory only, and a precision-first one (#9192): a page is listed because it names a
symbol, wire route or SDK method this diff touched — not because it mentions a changed
package. Each row says which anchor put it there, so a wrong row is reportable rather than
merely annoying. To re-verify, run the docs-accuracy-audit workflow scoped to these files:
node scripts/docs-audit/affected-docs.mjs a7c14cdf371bdb0f22be28e47a40fab9f7be88d0 → pass the list as
args.docs, on the commit named under Which tree this was computed on.

os-trump and others added 3 commits September 7, 2026 11:32
Patch round 1 on the contract review. settle() refuses succeeded -> failed in
both stores and in the engine's in-process fallback: a forced replay that throws
must not rewrite a delivered window, because that silently reopens the unforced
re-delivery door this card exists to close. failed -> succeeded stays required.

The two headers that recorded the immutability retirement stated a rule the code
did not hold; both now state the rule as implemented. Barrel surface narrowed to
what has consumers, and ReplayGuard is nameable from service-job.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012zTkyNHJ7TkuN2oXtP5x37
…icate

It restates update() rather than passing the base double's through, so it is a
double in its own right and carries the predicate itself.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012zTkyNHJ7TkuN2oXtP5x37
os-trump and others added 2 commits September 7, 2026 12:21
Text only; no behaviour change and no new pins.

- settle()'s absorbing rule is enforced read-then-write against the persisted
  store, so it is not atomic. The window is named rather than closed: the
  engine's by-id update cannot express a conditional write.
- the replay-pass residue comment claimed a later fire clears an abandoned
  pass. It does not -- the handler deletes only on a key match -- so the pass
  outlives every fire in every other window, inert throughout.
- the readDispatch degradation warning described the both-absent ledger while
  reading as if it covered a store with settle() but no read(), which is
  weaker still.
- the once-schedule changeset line asserted an unmeasured restart behaviour
  instead of the measured consequence: a replay before the due instant claims
  the single window, so the real fire then no-ops.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012zTkyNHJ7TkuN2oXtP5x37
@os-trump
os-trump marked this pull request as ready for review September 7, 2026 13:45
@os-trump
os-trump enabled auto-merge September 7, 2026 13:45
@os-trump
os-trump added this pull request to the merge queue Sep 7, 2026
Merged via the queue into main with commit 8341ed2 Sep 7, 2026
43 checks passed
@os-trump
os-trump deleted the claude/issue-14501-schedule-dispatch-claim-ledger branch September 7, 2026 14:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dependencies Pull requests that update a dependency file documentation Improvements or additions to documentation size/xl tests tooling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[finding] A schedule (cron) flow has no dispatch-claim ledger while a time_relative flow does — a re-run of a digest re-notifies

2 participants