PR 2: BashLexer + opaque-region scanner

IMPLEMENTATION_PLAN.md

@@ -42,14 +42,26 @@ bulldoze priorities.
 - [x] Update SPEC.md §3 to enumerate `Arg.IsCwdAttribution`; cross-tfm
       note on `VerbChain.Joined`
 
-### 3. BashLexer (SPEC §5)
-
-- [ ] Token kinds: WORD, QUOTED_STRING, OPERATOR, WHITESPACE,
-      CONTINUATION
-- [ ] Quote handling (single literal, double with `\"`, `\\`, `\$`)
-- [ ] Escape handling outside quotes
-- [ ] Operator boundaries (no whitespace required)
-- [ ] Heavy unit tests on tokenization
+### 3. BashLexer + opaque-region scanner — PR 2, in progress
+
+- [x] `Internal/Lexing/OpaqueRegionScanner.cs` — shared, grammar-agnostic;
+      `Scan` for `(`/`)` style + `ScanSymmetric` for backtick; quote-aware,
+      escape-aware, nesting-aware
+- [x] `Internal/Bash/Lexing/{BashLexer,BashToken,BashTokenKind}.cs`
+- [x] Token kinds: Word, QuotedString, Operator, Whitespace, Continuation,
+      OpaqueSubstitution, UnparseableSentinel
+- [x] Quote handling (single literal, double with `\"`, `\\`, `\$`,
+      `\` + newline)
+- [x] Escape handling outside quotes
+- [x] Operator boundaries (no whitespace required); `<<-` heredoc variant
+- [x] `$(…)` and backticks → `OpaqueSubstitution` (locked interpretation #2)
+- [x] `$((expr))` and `${var//pat/repl}` → `UnparseableSentinel`
+- [x] Heredoc body skip per SPEC §4
+- [x] 78 lexer + scanner unit tests (combined with PR 1's 18 → 96/96
+      passing)
+- [x] SPEC §1 / §5 / §11 updated for token kinds + non-goal additions
+- [x] OpenSpec change `v0.1-locked-interpretations` tasks.md updated
+      (Phase 2 marked [x])
 
 ### 4. Verb tables (SPEC §6, data only)
 

SPEC.md

@@ -48,6 +48,12 @@ command can consume it.
 - Variable expansion. We mark dynamic tokens, never resolve them.
 - Function definitions, here-docs body extraction, complex parameter
   expansion (`${var//pattern/replacement}`), arithmetic expansion.
+- Command-substitution evaluation. `$(cmd)` and backtick `` `cmd` `` are
+  recognized at the lex level and collapsed into a single
+  `Kind=DynamicSkip, IsPath=false` arg per locked interpretation #2 (see
+  `openspec/changes/archive/.../v0.1-locked-interpretations`). The
+  surrounding clause stays parseable so hard-deny rules still fire on
+  visible parts.
 - Performance tuning beyond "fast enough to invoke per shell call without
   noticeable latency" (~1ms per typical input).
 
@@ -359,14 +365,28 @@ quoted_string   := single-quoted | double-quoted
 The lexer produces tokens consumed by the parser. Token kinds:
 
 - **WORD** — sequence of non-whitespace, non-operator, non-quote chars.
-  Example: `git`, `/etc/foo`, `--force`, `~/path`, `$VAR`.
+  Example: `git`, `/etc/foo`, `--force`, `~/path`, `$VAR`. Simple
+  parameter expansion `${VAR}` (no `//` slash) is absorbed into a Word
+  token; the resolver in §8 decides `Kind`.
 - **QUOTED_STRING** — single- or double-quoted string. The lexer strips
   the quote delimiters from the token value. Example: `"hello world"`
   becomes the token value `hello world`.
 - **OPERATOR** — `&&`, `||`, `;`, `|`, `>`, `>>`, `<`, `2>`, `2>>`,
-  `(`, `)`, `<<`.
-- **WHITESPACE** — one or more spaces or tabs. Discarded after splitting.
+  `(`, `)`, `<<`, `<<-`.
+- **WHITESPACE** — one or more spaces or tabs (or newlines outside a
+  heredoc body). Discarded after splitting.
 - **CONTINUATION** — `\` + `\n`. Treated as whitespace.
+- **OPAQUE_SUBSTITUTION** — `$(cmd)` or backtick `` `cmd` ``. The full
+  substitution slice (including delimiters) becomes a single token.
+  Boundary tracking handles nested same-kind regions, nested quotes,
+  and `\X` escapes via a shared opaque-region scanner. The parser
+  consumes this token as `Arg{ Kind=DynamicSkip, IsPath=false,
+  Resolved=null }` per locked interpretation #2.
+- **UNPARSEABLE_SENTINEL** — `$((expr))` arithmetic expansion or
+  `${var//pat/repl}` complex parameter expansion. The lexer skips past
+  the matching close (`))` or `}` respectively) and emits a sentinel
+  whose reason names the rejected construct. The parser consumes this
+  token by setting outer `ParsedCommand.IsUnparseable = true` (see §11).
 
 ### Quote handling
 
@@ -759,7 +779,11 @@ Conditions that produce `IsUnparseable = true`:
   `then`, `fi`, `case`, `esac`).
 - Function definitions (`name() { ... }`).
 - Process substitution (`<(cmd)`, `>(cmd)`).
-- Recursion depth exceeded on `bash -c` chains.
+- Arithmetic expansion `$((expr))` (per §1 non-goal; lexer emits an
+  UNPARSEABLE_SENTINEL token; parser sets the outer flag).
+- Complex parameter expansion `${var//pat/repl}` (per §1 non-goal; same
+  mechanism).
+- Recursion depth exceeded on `bash -c` chains (>5 levels).
 
 Consumers (e.g. Netclaw's gate evaluator) route unparseable commands to a
 safe-fail path (prompt the user; offer only Once and Deny — no persistent

openspec/changes/v0.1-locked-interpretations/tasks.md

@@ -20,36 +20,46 @@ the SPEC.md sections that get updated alongside the implementation.
 - [x] 1.6 Add both projects to `ShellSyntaxTree.slnx`
 - [x] 1.7 Bootstrap OpenSpec scaffolding (`openspec/`, change directories,
       this proposal/design/tasks/specs delta)
-- [ ] 1.8 Update `SPEC.md` §2/§3 to enumerate `Arg.IsCwdAttribution`;
+- [x] 1.8 Update `SPEC.md` §3 to enumerate `Arg.IsCwdAttribution`;
       annotate `VerbChain.Joined` to use `string.Join(" ", Tokens)` for
       cross-tfm compatibility (was `string.Join(' ', …)`, char overload
       missing on netstandard2.0)
-- [ ] 1.9 Update `IMPLEMENTATION_PLAN.md` — mark PR 1 in-progress;
+- [x] 1.9 Update `IMPLEMENTATION_PLAN.md` — mark PR 1 in-progress;
       reference this OpenSpec change
-- [ ] 1.10 Update `TOOLING.md` to list installed OpenSpec skills under
+- [x] 1.10 Update `TOOLING.md` to list installed OpenSpec skills under
        Helper Skills
-- [ ] 1.11 Run `pwsh ./scripts/Add-FileHeaders.ps1`; verify with `-Verify`
-- [ ] 1.12 `dotnet build -c Release` clean; `dotnet test -c Release`
-       all-green
-- [ ] 1.13 Commit (signed) and push `pr1-bootstrap`; open PR with
-       `gh pr merge --auto --squash` against `dev`
-- [ ] 1.14 On merge: archive this OpenSpec change to
+- [x] 1.11 Run `pwsh ./scripts/Add-FileHeaders.ps1`; verify with `-Verify`
+- [x] 1.12 `dotnet build -c Release` clean; `dotnet test -c Release`
+       all-green (18/18 tests passing)
+- [x] 1.13 Commit (signed) and push `pr1-bootstrap`; opened PR #4 with
+       `gh pr merge --auto --squash` against `dev`. Auto-merged at
+       2026-05-10T17:55:09Z (Linux 27s, Windows 1m19s).
+- [ ] 1.14 Archive this OpenSpec change to
        `openspec/changes/archive/2026-05-10-v0.1-locked-interpretations/`
+       (deferred: not all interpretations have landed yet — change spans
+       PRs 1–6. Archive when PR 6 merges.)
 
 ## 2. PR 2 — Lexer + opaque-region scanner (interpretation #2)
 
-- [ ] 2.1 Implement `Internal/Lexing/OpaqueRegionScanner.cs` — finds
-      balanced delimiters with quote-aware nesting; configurable open/close
-- [ ] 2.2 Implement `Internal/Bash/Lexing/BashLexer.cs` per SPEC §5
-- [ ] 2.3 Recognize `$(...)` and `` ` `` regions via the scanner; emit
-      `OpaqueSubstitution` token
-- [ ] 2.4 Recognize `$((` and `${var//` openers; emit `IsUnparseable`
-      sentinel
-- [ ] 2.5 Update `SPEC.md` §1 (note that command substitution is marked
-      DynamicSkip), §5 (lex rules for opaque regions), §11 (add arithmetic
-      and complex-param-expansion to IsUnparseable triggers)
-- [ ] 2.6 Open OpenSpec change `bash-lexer-opaque-regions` capturing the
-      §1/§5/§11 deltas in their final form
+- [x] 2.1 Implement `Internal/Lexing/OpaqueRegionScanner.cs` — finds
+      balanced delimiters with quote-aware nesting; `Scan` for `(`/`)`
+      style and `ScanSymmetric` for backtick style; honors `\X` escapes,
+      single-quote literal preservation, double-quote escape table
+- [x] 2.2 Implement `Internal/Bash/Lexing/BashLexer.cs` per SPEC §5;
+      single entry point `Tokenize(string) -> IReadOnlyList<BashToken>`
+- [x] 2.3 Recognize `$(...)` and backtick `` `...` `` regions via the
+      scanner; emit `OpaqueSubstitution` token
+- [x] 2.4 Recognize `$((` and `${var//` openers; emit
+      `UnparseableSentinel` token with reason naming the construct
+- [x] 2.5 Update `SPEC.md` §1 non-goals (command substitution → DynamicSkip),
+      §5 (add OPAQUE_SUBSTITUTION + UNPARSEABLE_SENTINEL token kinds, add
+      `<<-` operator, simple `${VAR}` absorbed into Word, newlines outside
+      heredoc treated as Whitespace), §11 (add arithmetic +
+      complex-param-expansion to IsUnparseable conditions list)
+- [ ] 2.6 Tests: 16 OpaqueRegionScanner tests + 62 BashLexer tests, all
+      green. Combined with PR 1's 18: 96/96 passing.
+- [ ] 2.7 Run header script; verify; commit (signed) and push `pr2-lexer`;
+      open PR with `gh pr merge --auto --squash`
 
 ## 3. PR 3 — Verb tables + parser core