[v0.3] Structured shell analysis and control-flow roadmap

[Aaronontheweb]

Problem

ShellSyntaxTree 0.2 exposes a flat ParsedCommand.Clauses projection. It
cannot faithfully represent loop headers, nested bodies, mutually exclusive
branches, repeated execution, bounded loop variables, or divergent shell
state. Unsupported forms fail closed, which is safe but creates material
approval fatigue for real agent-authored commands.

The Netclaw 0.25.3 redirect incident also demonstrated a boundary problem:
static descriptor operations and genuinely dynamic values can both appear as
DynamicSkip, forcing consumers to reconstruct shell grammar from raw text.

v0.3 outcome

Add an evolutionary, security-oriented model with:

• a strongly typed syntax tree above existing Clause leaves;
• a library-owned Commands projection containing every command that may
  execute, including conditions, iterators, substitutions, branches, and loop
  bodies;
• exact, finite, bounded-pattern, and unknown shell value domains;
• conservative cwd and variable-state joins;
• explicit redirect operations and target facts;
• a conservative ParsedCommand.Clauses compatibility projection;
• separate Bash and PowerShell grammar front ends feeding shared analysis only
  where both implementations prove the same abstraction.

The security invariant is:

Identify every command that may execute and derive no narrower authorization
scope than shell semantics can prove.

Boundaries

• Do not execute commands, enumerate globs, inspect runtime variables, or use a
  live shell to derive authorization facts.
• Do not build a shared lexer or shared parser base class.
• Do not embed Git, rm, Docker, cmdlet, or other executable-specific option and
  operand grammars in ShellSyntaxTree.
• Do not classify URLs or environment assignments as harmless globally;
  executable-aware consumers own those decisions.
• Do not treat heredoc bodies as commands merely because a receiving executable
  might interpret the data as code.
• Unknown or incomplete executable regions remain fail closed.

Delivery order

1. Lock the public API, compatibility projection, completeness rules, fixed
   analysis bounds, and Bash/PowerShell supported-construct matrix.
2. Land the behavior-preserving shared native-fragment extraction in Extract shared native argument fragment classification #69.
3. Produce Syntax, Commands, and unchanged compatibility Clauses for the
   existing grammar.
4. Add explicit redirect semantics and migrate the Netclaw consumer path.
5. Add Bash for ... in as the first end-to-end control-flow slice.
6. Add PowerShell foreach as the second slice before expanding all Bash
   constructs.
7. Extract shared occurrence, value-domain, and state-join machinery proven by
   those two working implementations.
8. Add condition loops and branches in separate shell-specific slices.
9. Preserve Bash heredocs with explicit delimiter, body, expansion, and completeness facts; add Bash <<< here strings; and gate process substitution, background lists, C-style loops, and arithmetic on their own executable-region and value-semantics designs.

Acceptance gates

• Every supported executable region appears exactly once in Commands.
• Conditions and iterators are never omitted from authorization traversal.
• Existing Clause, Arg, Redirect, ClauseElement, and VerbChain facts
  remain available.
• Old consumers receive a conservative flattened view and do not silently miss
  nested commands.
• Exact, finite, pattern, and unknown facts are never conflated.
• Static and computed redirect targets are distinguishable without parsing raw
  prefixes.
• Unsupported execution-bearing regions set IsUnparseable=true; partial
  results are diagnostic only.
• Both shell corpora, public API snapshots, PII audit, Linux/Windows CI, and
  Netclaw integration matrices pass.

Related work

• Support structured control flow (loops) in Bash and PowerShell grammars #71 — structured control-flow request and production motivation
• Extract shared native argument fragment classification #69 — narrow shared native argument-fragment refactoring
• fix(approvals): stop treating fd-dup redirects (2>&1) as dynamic syntax netclaw-dev/netclaw#1776 — temporary Netclaw 0.25.4 fd-redirect workaround
• Repository design: openspec/changes/v0-3-structured-shell-analysis/

The OpenSpec contains the proposal, architectural decisions, five capability
specifications, and the ordered implementation checklist. Public type names
and analysis caps remain explicit contract-lock tasks; production code should
not begin until those are reviewed.

[Aaronontheweb]

Scope update: v0.3 explicitly includes updating docs/CONSUMER_GUIDE.md and the README usage and migration examples. The OpenSpec proposal and task group 5 now treat those user-facing documents as required consumer-migration deliverables, not release cleanup.

[Aaronontheweb]

Design-corpus and contract-lock work is now in #74.

The paired corpus currently has 19 Bash and 18 PowerShell cases and validates recorded v0.2 behavior, desired structure/occurrences, bounded values, redirects, compatibility, security invariants, PII safety, and real-shell parse validity. The review locked these previously open decisions:

• closed strongly typed syntax-record family; authorization uses the complete Commands projection
• ImmediateRole plus compositional ancestry
• structural completeness independent from value precision
• fixed limits: 32 value candidates, 16 structural containers, existing wrapper depth 5
• identical in-memory Clause instance across syntax, occurrence, and compatibility projections
• static Bash glob cover only; divergent cwd immediately becomes Unknown
• unparseable results have empty Commands and Clauses; partial Syntax is diagnostics only
• separate Bash and PowerShell grammar matrices and PowerShell authored-binding versus effective-value semantics
• existing Bash heredocs preserved with explicit delimiter/body/expansion/completeness facts; Bash <<< here strings included; process substitution, background lists, case/switch, arithmetic/C-style loops, and definitions remain independently gated
• consumer-guide migration requirements are now explicit, including equality and persistence caveats

The remaining contract tasks 1.6-1.7 intentionally move with the production API + public snapshot PR so SPEC.md, SPEC.POWERSHELL.md, and the assembly change atomically.