Skip to content

Scatter/gather workers are not jobs — invisible, unaddressable, no progress #246

Description

@tobert

Found by a job-system design audit for embedder/MCP use (2026-07-30); confirmed by grep and by a second model family.

crates/kaish-kernel/src/scheduler/scatter.rs contains zero references to JobManager. Workers are fork_attached() kernels (scatter.rs:259) driven by bare tokio::spawn handles (scatter.rs:311).

Consequences — scatter workers are:

  • absent from jobs and from /v/jobs/,
  • not individually addressable or cancellable (kill %N has nothing to target),
  • entirely synchronous from the caller's view: a scatter … | gather produces nothing observable until the whole fan-out returns inside one execute() call.

Why an embedder hits it

Parallel fan-out is exactly the shape of work an MCP server wants to expose as long-running and pollable — "analyze these 40 files, poll me for progress". Today kaijutsu can only offer it as one blocking execute() that either finishes or times out, with zero progress visibility and no way to cancel one straggler.

This is arguably a bigger latent gap for kaijutsu's direction than the live-output issue. It's ranked below it only because live output is a broken promise while this is an unmade one.

Two independent options

(a) Register each worker as a child job with a parent job id, so /v/jobs shows the fan-out. Makes the whole scatter observable and per-worker killable. More machinery — needs a parent/child relationship on JobInfo.

(b) Let a whole scatter | gather pipeline be backgrounded as one job with progress (n/total) surfaced on JobInfo. Much cheaper, probably 80% of the value.

Recommend costing (b) first.

Size: design session. Not breaking (additive).

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions