fix(orchestrations): bound loop / sub_orchestration nesting depth

[arantespp]

Fixes #1185.

The gap

Nothing bounded how deep loop / sub_orchestration nesting went. A graph whose sub_orchestration node named itself — directly, or through a cycle of two graphs — started a child run that started a child run, until something incidental stopped it. None of the platform's existing bounds reaches that path, for the reasons the issue's table sets out.

The bound

Every run now carries a run_depth: 0 for a caller-started run, one more than its parent's for a loop / sub_orchestration child. Read off the parent's row by the engine driving it and passed through NestedRunParent, rather than looked up from parent_run_id — the row is already in hand there, and a child whose parent row had since been deleted would otherwise restart the count from zero.

startOrchestrationRun is the single choke point both node types already go through, so the check lives there — before the child's row exists, so a refusal leaves nothing queued behind it, and any nested-run caller added later inherits the bound.

The effective bound is the smaller of:

Bound	Where	Default
MAX_ORCHESTRATION_RUN_DEPTH	deployment env var	10
max_run_depth	new project column, PATCH /api/v1/projects/{project_id}	null (defer to the deployment's)

Both read at spawn time, not pinned on the tree's root, so lowering the number stops a tree that is already recursing.

The silent stop it had to fix too

The refusal alone would not have surfaced. A child run catches its own failures — executeRun settles the run rather than letting the throw escape — so the seam handed the parent a settled failed run and the node took output ?? {} from it and carried on. With the guard in place but nothing propagating, all four bound tests reported the root run as succeeded (verified in this session, red).

So a child settling in a non-success terminal status (failed, cancelled, expired) now fails the node that started it — the same rule a workflow on_enter dispatch already follows (ORCHESTRATION_DISPATCH_FAILED) — and under the child's own error code where it has one, so the cause reaches the run a caller reads instead of stopping at the run that noticed it. error.meta names the child run and node at each level, so parent_orchestration_run_id still walks down to the failure. A child that has merely parked (awaiting_input, sleeping) has not settled and is unaffected.

This is a behaviour change beyond the depth bound, and the reviewer's call: today a sub_orchestration child that fails is silently swallowed and the parent succeeds on an empty artifact. Acceptance criteria 1, 2 and 4 cannot hold without it.

Acceptance

• A self-referencing sub_orchestration graph terminates with an error naming the depth bound — ORCHESTRATION_RUN_DEPTH_LIMIT, "would reach nesting depth 11, past the limit of 10", on the run the caller started
• The same for a cycle spanning two graphs, and for loop nodes
• A legitimate deep composition (6 levels) is unaffected under the default
• Observable: run_depth on every run, plus the failed run — which the existing orchestration_runs.failed listener already auto-files as a run_failed exception carrying { code, message }, so no new event was needed

Red/green

Confirmed red before green, both halves separately:

Neutered	Result
depth guard + propagation	the three recursion tests never terminate (timed out at 120s) — the reported bug; the project-bound test reports succeeded
propagation only	all four bound tests report succeeded — the silent stop
neither	11/11 green

Also in this PR

• ORCHESTRATION_RUN_DEPTH_LIMIT (409) and ORCHESTRATION_NESTED_RUN_FAILED (422) in the registry, both with ERROR_RESOLUTIONS hints
• run_depth on the run schema; max_run_depth on the project schema, request body and docs
• orchestrations.md — Nesting depth, A child run's failure fails its parent, the two error rows, the env var; projects.md — the new field
• tests/smoke-tests.sh — the project bound set/cleared, run_depth on a child, and a self-referencing graph terminating on the bound

Resolved open questions

Q: Opt-in with an unbounded default, or a finite default?
A: Finite (10) — resolved by long-term; checked: it is the house pattern for every
   comparable bound in this repo (max_call_depth 10, TASK_AUTOMATION_CHAIN_LIMIT 50,
   MAX_CONTINUATION_CHAIN_GENERATIONS 100), and an opt-in default enforces nothing.
   10 matches max_call_depth, the closest analog (synchronous recursion started by a
   resource naming another). Verified a 6-level composition passes under it.

Q: Where is it configured — per node, per orchestration, per project, or per deployment?
A: Deployment env var + project column, min-of-the-two — resolved by long-term;
   checked: it is byte-for-byte the shape #1176 used for maxChainGenerations
   (resolveEffectiveLimit in generationChain.ts), so a reader who knows one knows
   this. A per-node field is purely additive later; a per-node bound alone would be
   opt-out, which the bound cannot be, since the graph that runs away is the one
   whose config is wrong.

Q: Bound depth, or bound the population of descendants the way maxChainGenerations does?
A: Depth — resolved by pareto; checked: the issue's acceptance criteria are
   depth-stated, and OrchestrationRun.parentRunId/parentNodeId already carry the
   identity, so a counter needs no walk and no new query. Documented honestly: this
   bounds recursion, not total work — a loop node fans out, so N children per level
   still permits N^depth runs, and width stays bounded by the node's `parallelism`
   and the project's `max_concurrent_runs`.

Q: Does a nested child's failure have to propagate, given the issue scopes only the bound?
A: Yes — resolved by long-term; checked empirically that without it the caller of a
   recursing tree is told `succeeded` (the four bound tests go red on exactly that),
   so criteria 1/2/4 are unreachable. Scoped to non-success terminal statuses and
   modelled on ORCHESTRATION_DISPATCH_FAILED rather than special-casing the depth
   code, which would have left the silent-stop hole open for every other failure.

Verification

• pnpm typecheck — clean (no as any / as unknown added)
• pnpm eslint — 36 errors before and after, all pre-existing max-lines on unrelated large test files; none in any file this PR touches
• pnpm test (server) — 238 suites / 6696 tests pass, every coverage threshold met
• pnpm --filter @soat/postgresdb test — 15/15, including the schema-drift suite against the two new columns
• pnpm run docs-lint — OK, 94 files
• pnpm --filter @soat/website test — 63/66; the same 3 fail on a clean main checkout (they need a built packages/website/build). The two that matter here — every registry code gets a section, and every emitted docs_url addresses a heading — pass.
• sh -n tests/smoke-tests.sh — POSIX-clean. The smoke and tutorials stacks were not run: they need Docker, which is unavailable in this environment. CI covers them.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Bh5kgHuqisvy4cLtMrNf4s

---

Generated by Claude Code

[github-actions]

Deploy Outputs

Package	Stack	Output Key	Output Value
@soat/website	SoatWebsite-claude-issue-1185-6qfhe7	BucketWebsiteURL	http://soatwebsite-claude-issue-1185-6qfhe7-staticbucket-ewacszr5zfxg.s3-website-us-east-1.amazonaws.com